Your Model Is Not Predicting Depression Well And That Is Why: A Case Study of PRIMATE Dataset

TL;DR

This study critically examines the PRIMATE dataset's annotation quality for depression detection, revealing issues with labels and proposing refined annotations by mental health experts to improve NLP model reliability.

Contribution

The paper highlights annotation validity concerns in mental health datasets and introduces expert reannotations for better depression and anhedonia detection.

Findings

Identification of false positives in PRIMATE annotations

Refined annotations improve dataset quality

Expert reannotation enhances model reliability

Abstract

This paper addresses the quality of annotations in mental health datasets used for NLP-based depression level estimation from social media texts. While previous research relies on social media-based datasets annotated with binary categories, i.e. depressed or non-depressed, recent datasets such as D2S and PRIMATE aim for nuanced annotations using PHQ-9 symptoms. However, most of these datasets rely on crowd workers without the domain knowledge for annotation. Focusing on the PRIMATE dataset, our study reveals concerns regarding annotation validity, particularly for the lack of interest or pleasure symptom. Through reannotation by a mental health professional, we introduce finer labels and textual spans as evidence, identifying a notable number of false positives. Our refined annotations, to be released under a Data Use Agreement, offer a higher-quality test set for anhedonia detection. This study underscores the necessity of addressing annotation quality issues in mental health datasets, advocating for improved methodologies to enhance NLP model reliability in mental health assessments.