Still Not Quite There! Evaluating Large Language Models for Comorbid Mental Health Diagnosis

TL;DR

This paper introduces ANGST, a new benchmark dataset for multi-label classification of depression and anxiety from social media posts, and evaluates state-of-the-art language models on this complex diagnostic task.

Contribution

The paper presents ANGST, the first benchmark for comorbid mental health diagnosis from social media, and provides a comprehensive evaluation of current language models on this challenging task.

Findings

GPT-4 outperforms other models but still has limited accuracy.

No model exceeds 72% F1 score in multi-label classification.

The task reveals significant challenges in applying language models to mental health diagnostics.

Abstract

In this study, we introduce ANGST, a novel, first-of-its kind benchmark for depression-anxiety comorbidity classification from social media posts. Unlike contemporary datasets that often oversimplify the intricate interplay between different mental health disorders by treating them as isolated conditions, ANGST enables multi-label classification, allowing each post to be simultaneously identified as indicating depression and/or anxiety. Comprising 2876 meticulously annotated posts by expert psychologists and an additional 7667 silver-labeled posts, ANGST posits a more representative sample of online mental health discourse. Moreover, we benchmark ANGST using various state-of-the-art language models, ranging from Mental-BERT to GPT-4. Our results provide significant insights into the capabilities and limitations of these models in complex diagnostic scenarios. While GPT-4 generally outperforms other models, none achieve an F1 score exceeding 72% in multi-class comorbid classification, underscoring the ongoing challenges in applying language models to mental health diagnostics.