MindSET: Advancing Mental Health Benchmarking through Large-Scale Social Media Data

TL;DR

MindSET is a large, high-quality social media benchmark dataset from Reddit, enabling improved mental health diagnosis detection and analysis through rigorous preprocessing and extensive annotations.

Contribution

This paper introduces MindSET, a new large-scale, cleaned, and annotated Reddit dataset for mental health analysis, surpassing previous benchmarks in size and quality.

Findings

Models trained on MindSET outperform previous benchmarks in diagnosis detection.

MindSET enables more accurate and diverse mental health analysis.

Linguistic analysis reveals psychological patterns across mental health conditions.

Abstract

Social media data has become a vital resource for studying mental health, offering real-time insights into thoughts, emotions, and behaviors that traditional methods often miss. Progress in this area has been facilitated by benchmark datasets for mental health analysis; however, most existing benchmarks have become outdated due to limited data availability, inadequate cleaning, and the inherently diverse nature of social media content (e.g., multilingual and harmful material). We present a new benchmark dataset, \textbf{MindSET}, curated from Reddit using self-reported diagnoses to address these limitations. The annotated dataset contains over \textbf{13M} annotated posts across seven mental health conditions, more than twice the size of previous benchmarks. To ensure data quality, we applied rigorous preprocessing steps, including language filtering, and removal of Not Safe for Work (NSFW) and duplicate content. We further performed a linguistic analysis using LIWC to examine psychological term frequencies across the eight groups represented in the dataset. To demonstrate the dataset utility, we conducted binary classification experiments for diagnosis detection using both fine-tuned language models and Bag-of-Words (BoW) features. Models trained on MindSET consistently outperformed those trained on previous benchmarks, achieving up to an \textbf{18-point} improvement in F1 for Autism detection. Overall, MindSET provides a robust foundation for researchers exploring the intersection of social media and mental health, supporting both early risk detection and deeper analysis of emerging psychological trends.