Exploring Large-Scale Language Models to Evaluate EEG-Based Multimodal Data for Mental Health

TL;DR

This study explores the use of large language models with multimodal data, including EEG, audio, and facial expressions, to improve mental health assessment through zero-shot and few-shot learning, showing multimodal integration enhances accuracy.

Contribution

It demonstrates the effectiveness of LLMs in multimodal mental health assessment, emphasizing the benefits of integrating EEG with other data modalities and few-shot learning.

Findings

Multimodal data improves mental health classification accuracy.

EEG integration with audio and images shows promising results.

1-shot learning outperforms zero-shot in this context.

Abstract

Integrating physiological signals such as electroencephalogram (EEG), with other data such as interview audio, may offer valuable multimodal insights into psychological states or neurological disorders. Recent advancements with Large Language Models (LLMs) position them as prospective ``health agents'' for mental health assessment. However, current research predominantly focus on single data modalities, presenting an opportunity to advance understanding through multimodal data. Our study aims to advance this approach by investigating multimodal data using LLMs for mental health assessment, specifically through zero-shot and few-shot prompting. Three datasets are adopted for depression and emotion classifications incorporating EEG, facial expressions, and audio (text). The results indicate that multimodal information confers substantial advantages over single modality approaches in mental health assessment. Notably, integrating EEG alongside commonly used LLM modalities such as audio and images demonstrates promising potential. Moreover, our findings reveal that 1-shot learning offers greater benefits compared to zero-shot learning methods.