Then and Now: Quantifying the Longitudinal Validity of Self-Disclosed Depression Diagnoses

TL;DR

This study examines how self-disclosed depression diagnoses on social media remain relevant over time, revealing their limitations and biases, and offers practical recommendations for improving mental health datasets.

Contribution

It provides the first longitudinal analysis of self-disclosed depression diagnoses, highlighting their dynamic nature and introducing methods to mitigate biases in mental health datasets.

Findings

Self-disclosed diagnoses often lose relevance over five years.

Personality biases influence mental health data curated from self-disclosures.

Recommendations include annotating diagnosis dates and controlling for biases.

Abstract

Self-disclosed mental health diagnoses, which serve as ground truth annotations of mental health status in the absence of clinical measures, underpin the conclusions behind most computational studies of mental health language from the last decade. However, psychiatric conditions are dynamic; a prior depression diagnosis may no longer be indicative of an individual's mental health, either due to treatment or other mitigating factors. We ask: to what extent are self-disclosures of mental health diagnoses actually relevant over time? We analyze recent activity from individuals who disclosed a depression diagnosis on social media over five years ago and, in turn, acquire a new understanding of how presentations of mental health status on social media manifest longitudinally. We also provide expanded evidence for the presence of personality-related biases in datasets curated using self-disclosed diagnoses. Our findings motivate three practical recommendations for improving mental health datasets curated using self-disclosed diagnoses: 1) Annotate diagnosis dates and psychiatric comorbidities; 2) Sample control groups using propensity score matching; 3) Identify and remove spurious correlations introduced by selection bias.