Large Language Models for Depression Recognition in Spoken Language Integrating Psychological Knowledge

TL;DR

This paper introduces a multimodal depression detection method combining large language models with audio features and psychological knowledge, improving diagnostic accuracy over previous approaches.

Contribution

It is the first to apply LLMs to multimodal depression detection, integrating psychological expertise and audio features for enhanced accuracy.

Findings

Improved MAE and RMSE in depression detection

Effective integration of psychological knowledge into LLMs

Utilization of Wav2Vec for audio feature extraction

Abstract

Depression is a growing concern gaining attention in both public discourse and AI research. While deep neural networks (DNNs) have been used for recognition, they still lack real-world effectiveness. Large language models (LLMs) show strong potential but require domain-specific fine-tuning and struggle with non-textual cues. Since depression is often expressed through vocal tone and behaviour rather than explicit text, relying on language alone is insufficient. Diagnostic accuracy also suffers without incorporating psychological expertise. To address these limitations, we present, to the best of our knowledge, the first application of LLMs to multimodal depression detection using the DAIC-WOZ dataset. We extract the audio features using the pre-trained model Wav2Vec, and mapped it to text-based LLMs for further processing. We also propose a novel strategy for incorporating psychological knowledge into LLMs to enhance diagnostic performance, specifically using a question and answer set to grant authorised knowledge to LLMs. Our approach yields a notable improvement in both Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) compared to a base score proposed by the related original paper. The codes are available at https://github.com/myxp-lyp/Depression-detection.git