When LLMs Meets Acoustic Landmarks: An Efficient Approach to Integrate Speech into Large Language Models for Depression Detection

TL;DR

This paper introduces a novel method that integrates acoustic landmarks into Large Language Models to improve depression detection from speech, achieving state-of-the-art results on the DAIC-WOZ dataset.

Contribution

The paper presents an innovative approach to incorporate acoustic landmarks into LLMs, enhancing multimodal depression detection from speech and text.

Findings

Achieved state-of-the-art results on DAIC-WOZ dataset.

Demonstrated the effectiveness of acoustic landmarks in depression detection.

Enhanced LLMs' ability to process speech signals for mental health analysis.

Abstract

Depression is a critical concern in global mental health, prompting extensive research into AI-based detection methods. Among various AI technologies, Large Language Models (LLMs) stand out for their versatility in mental healthcare applications. However, their primary limitation arises from their exclusive dependence on textual input, which constrains their overall capabilities. Furthermore, the utilization of LLMs in identifying and analyzing depressive states is still relatively untapped. In this paper, we present an innovative approach to integrating acoustic speech information into the LLMs framework for multimodal depression detection. We investigate an efficient method for depression detection by integrating speech signals into LLMs utilizing Acoustic Landmarks. By incorporating acoustic landmarks, which are specific to the pronunciation of spoken words, our method adds critical dimensions to text transcripts. This integration also provides insights into the unique speech patterns of individuals, revealing the potential mental states of individuals. Evaluations of the proposed approach on the DAIC-WOZ dataset reveal state-of-the-art results when compared with existing Audio-Text baselines. In addition, this approach is not only valuable for the detection of depression but also represents a new perspective in enhancing the ability of LLMs to comprehend and process speech signals.