Recent studies have shown that machine learning can identify individuals with mental illnesses by analyzing their social media posts. Topics and words related to mental health are some of the top predictors. These findings have implications for early detection of mental illnesses. However, they also raise numerous privacy concerns. To fully evaluate the implications for privacy, we analyze the performance of different machine learning models in the absence of tweets that talk about mental illnesses. Our results show that machine learning can be used to make predictions even if the users do not actively talk about their mental illness. To fully understand the implications of these findings, we analyze the features that make these predictions possible. We analyze bag-of-words, word clusters, part of speech n-gram features, and topic models to understand the machine learning model and to discover language patterns that differentiate individuals with mental illnesses from a control group. This analysis confirmed some of the known language patterns and uncovered several new patterns. We then discuss the possible applications of machine learning to identify mental illnesses, the feasibility of such applications, associated privacy implications, and analyze the feasibility of potential mitigations.
 Mohammed Al-Mosaiwi and Tom Johnstone. In an absolute state: elevated use of absolutist words is a marker specific to anxiety, depression and suicidal ideation. Clinical Psychological Science, January 2018.
 Nazanin Andalibi, Pinar Ozturk, and Andrea Forte. Sensitive Self-disclosures, Responses, and Social Support on Instagram: The Case of #Depression. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing, CSCW ’17, pages 1485–1500, New York, NY, USA, 2017. ACM.
 Athanasios Andreou, Marcio Silva, Fabrício Benevenuto, Oana Goga, Patrick Loiseau, and Alan Mislove. Measuring the Facebook Advertising Ecosystem. In NDSS 2019 - Proceedings of the Network and Distributed System Security Symposium, San Diego, United States, February 2019.
 Stefan Axelsson. The Base-rate Fallacy and Its Implications for the Difficulty of Intrusion Detection. In Proceedings of the 6th ACM Conference on Computer and Communications Security, CCS ’99, pages 1–7, New York, NY, USA, 1999. ACM.
 Steven Bird, Ewan Klein, and Edward Loper. Natural Language Processing with Python. O’Reilly Media, Inc., 1st edition, 2009.
 Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. Class-based N-gram Models of Natural Language. Comput. Linguist., 18(4):467–479, December 1992.
 Glen Coppersmith, Mark Dredze, and Craig Harman. Quantifying Mental Health Signals in Twitter. Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 51–60, 2014.
 Glen Coppersmith, Mark Dredze, Craig Harman, and Kristy Hollingshead. From ADHD to SAD: Analyzing the Language of Mental Health on Twitter through Self-Reported Diagnoses. Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 1–10, 2015.
 Glen Coppersmith, Mark Dredze, Craig Harman, Kristy Hollingshead, and Margaret Mitchell. CLPsych 2015 Shared Task: Depression and PTSD on Twitter. the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 31–39, 2015.
 Glen Coppersmith, Craig Harman, and Mark Dredze. Measuring Post Traumatic Stress Disorder in Twitter. In Proceedings of the 7th International AAAI Conference on Weblogs and Social Media (ICWSM)., 2(1):23–45, 2014.
 Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. Predicting Depression via Social Media. In Proceedings of the 7th International AAAI Conference on Weblogs and Social Media, pages 128–138, 2013.
 Olive Jean Dunn. Multiple comparisons among means. Journal of the American Statistical Association, 56(293):52–64, 1961.
 Kevin Gimpel, Nathan Schneider, Brendan O’Connor, Dipanjan Das, Daniel Mills, Jacob Eisenstein, Michael Heilman, Dani Yogatama, Jeffrey Flanigan, and Noah A. Smith. Part-of-speech Tagging for Twitter: Annotation, Features, and Experiments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers - Volume 2, HLT ’11, pages 42–47, Stroudsburg, PA, USA, 2011. Association for Computational Linguistics.
 Su Golder, Shahd Ahmed, Gill Norman, and Andrew Booth. Attitudes toward the ethics of research using social media: A systematic review, June 2017.
 Sharath Chandra Guntuku, David B. Yaden, Margaret L. Kern, Lyle H. Ungar, and Johannes C. Eichstaedt. Detecting depression and mental illness on social media: an integrative review, 2017.
 Seda Gurses, Rebekah Overdorf, and Ero Balsa. POTs: The revolution will not be optimized? 11th Hot Topics in Privacy Enhancing Technologies (HotPETs), 2018.
 Melinda R Hess and Jeffrey D Kromrey. Robust confidence intervals for effect sizes: A comparative study of cohen’sd and cliff’s delta under non-normality and heterogeneous variances. 2004.
 Alvarez-Conrad Jennifer, Zoellner Lori A., and Foa Edna B. Linguistic predictors of trauma pathology and physical health. Applied Cognitive Psychology, 15(7):S159–S170, 2001.
 Ron Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the 14th International Joint Conference on Artificial Intelligence - Volume 2, IJCAI’95, pages 1137–1143, San Francisco, CA, USA, 1995. Morgan Kaufmann Publishers Inc.
 Björn Lantz. The large sample size fallacy. Scandinavian journal of caring sciences, 27(2):487–492, 2013.
 Naomi Lee. Trouble on the radar. The Lancet, 384(29):1917, 2014.
 Huina Mao, Xin Shuai, and Apu Kapadia. Loose Tweets: An Analysis of Privacy Leaks on Twitter. In Proceedings of the 10th Annual ACM Workshop on Privacy in the Electronic Society, WPES ’11, pages 1–12, New York, NY, USA, 2011. ACM.
 Jon D Mcauliffe and David M Blei. Supervised topic models. In Advances in neural information processing systems, pages 121–128, 2008.
 Jude Mikal, Samantha Hurst, and Mike Conway. Ethical issues in using Twitter for population-level depression monitoring: A qualitative study. BMC Medical Ethics, 17(1), 2016.
 Nilly Mor and Jennifer Winquist. Self-Focused Attention and Negative Affect : A Meta-Analysis. Psychological bulletin, 128(4):638–662, 2002.
 Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, and Nathan Schneider. Part-of-speech tagging for twitter: Word clusters and other advances. 2012.
 Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277, 2016.
 James W Pennebaker and Laura A King. Linguistic styles: Language use as an individual difference. Journal of personality and social psychology, 77(6):1296, 1999.
 Daniel Preoţiuc-Pietro, Johannes Eichstaedt, Gregory Park, Maarten Sap, Laura Smith, Victoria Tobolsky, H Andrew Schwartz, and Lyle Ungar. The role of personality, age, and gender in tweeting about mental illness. In Proceedings of the 2nd Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 21–30, 2015.
 Philip Resnik, William Armstrong, Leonardo Claudino, Thang Nguyen, Viet-An Nguyen, and Jordan Boyd-graber. Beyond LDA : Exploring Supervised Topic Modeling for Depression-Related Language in Twitter. Proceedings of the 52nd Workshop Computational Linguistics and Clinical Psychology, 1(2014):99–107, 2015.
 Philip Resnik, William Armstrong, Leonardo Claudino, Thang Nguyen, Viet-An Nguyen, and Jordan Boyd-Graber. The University of Maryland CLPsych 2015 Shared Task System. Proceedings of the Workshop on Computational Linguistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pages 54–60, 2015.
 Philip Resnik, Anderson Garron, and Rebecca Resnik. Using Topic Modeling to Improve Prediction of Neuroticism and Depression. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1348–1353, 2013.
 Beatrice Santorini. Part-of-speech tagging guidelines for the Penn Treebank Project (3rd revision). Technical Reports (CIS), page 570, 1990.
 H Andrew Schwartz, Johannes C Eichstaedt, Margaret L Kern, Lukasz Dziurzynski, Stephanie M Ramones, Megha Agrawal, Achal Shah, Michal Kosinski, David Stillwell, Martin E.P. Seligman, and Lyle H Ungar. Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach. PLoS ONE, 8(9), 2013.
 Manya Sleeper, Justin Cranshaw, Patrick Gage Kelley, Blase Ur, Alessandro Acquisti, Lorrie Faith Cranor, and Norman Sadeh. “I read my Twitter the next morning and was astonished” a conversational perspective on Twitter regrets. Chi, pages 3277–3286, 2013.
 Daria Smirnova, Paul Cumming, Elena Sloeva, Natalia Kuvshinova, Dmitry Romanov, and Gennadii Nosachev. Language patterns discriminate mild depression from normal sadness and euthymic state. Frontiers in Psychiatry, 9:105, 2018.
 Gail M Sullivan and Richard Feinn. Using effect size—or why the p value is not enough. Journal of graduate medical education, 4(3):279–282, 2012.
 Yla R Tausczik and James W Pennebaker. The psychological meaning of words: Liwc and computerized text analysis methods. Journal of language and social psychology, 29(1):24–54, 2010.
 The Federal Trade Comission. Data brokers: A call for transparency and accountability. Data Brokers and the Need for Transparency and Accountability, pages 1–101, 2014.
 Rianne Van der Zanden, Keshia Curie, Monique Van Londen, Jeannet Kramer, Gerard Steen, and Pim Cuijpers. Web-based depression treatment: Associations of clients’ word use with adherence and outcome. Journal of Affective Disorders, 160:10–13, 2014.
 Yang Wang, Gregory Norcie, Saranga Komanduri, Alessandro Acquisti, Pedro Giovanni Leon, and Lorrie Faith Cranor. “I Regretted the Minute I Pressed Share”: A Qualitative Study of Regrets on Facebook. In Proceedings of the Seventh Symposium on Usable Privacy and Security, SOUPS ’11, pages 10:1–10:16, New York, NY, USA, 2011. ACM.
 Johannes Zimmermann, Timo Brockmeyer, Matthias Hunn, Henning Schauenburg, and Markus Wolf. First-person Pronoun Use in Spoken Language as a Predictor of Future Depressive Symptoms: Preliminary Evidence from a Clinical Sample of Depressed Patients. Clinical Psychology & Psychotherapy, 24(2):384–391, mar 2017.