FedMental: Evaluating Federated Learning for Mental Health Detection from Social Media Data

TL;DR

This paper evaluates federated learning and Differentially Private FL for mental health detection from social media, showing comparable results to centralized methods but significant privacy-performance trade-offs.

Contribution

It provides a comprehensive empirical assessment of privacy-preserving ML techniques for mental health prediction on social media data.

Findings

Federated learning achieves near-centralized performance in depression detection.

Differentially Private FL incurs large accuracy drops, highlighting privacy-performance trade-offs.

Highly informative linguistic markers are distorted under privacy-preserving techniques.

Abstract

Social media text data are often used to train Machine Learning (ML) models to identify users exhibiting high-risk mental health behaviors. However, sharing this sensitive data poses privacy risks and limits the growth of benchmark datasets. We comprehensively evaluate whether privacy-preserving ML techniques can enable safer data sharing while preserving performance. Specifically, we apply federated learning (FL) and Differentially Private FL for two widely-studied mental health prediction tasks: depression detection on X (Twitter) and suicide crisis detection on Reddit. We simulate realistic data-sharing scenarios by treating each user as a client in a non-IID setting, evaluating across different client fractions, aggregation strategies, and privacy budgets. While FL achieves comparable performance to centralized training (centralized F1 = 85.63; best FL model F1 = 83.16) on depression identification, we find that Differentially Private FL has a large performance-privacy trade-off (up to F1 = 27.01 drop) even with low levels of noise (epsilon = 50). This is due to the distortion of highly informative yet sparse mental health linguistic markers related to mental health, like health topics and emotion words. This research empirically demonstrates the potential and limitations of current privacy preservation techniques for mental health inference tasks.