MMFformer: Multimodal Fusion Transformer Network for Depression Detection

TL;DR

MMFformer is a novel multimodal transformer network that effectively detects depression from social media data by capturing spatial and temporal features across multiple modalities, outperforming existing methods.

Contribution

The paper introduces MMFformer, a new multimodal fusion transformer architecture that enhances depression detection accuracy by integrating spatial and temporal features from videos and audio.

Findings

Surpasses state-of-the-art with 13.92% F1-Score improvement on D-Vlog dataset.

Achieves 7.74% F1-Score increase on LMVD dataset.

Effectively fuses multimodal social media data for depression analysis.

Abstract

Depression is a serious mental health illness that significantly affects an individual's well-being and quality of life, making early detection crucial for adequate care and treatment. Detecting depression is often difficult, as it is based primarily on subjective evaluations during clinical interviews. Hence, the early diagnosis of depression, thanks to the content of social networks, has become a prominent research area. The extensive and diverse nature of user-generated information poses a significant challenge, limiting the accurate extraction of relevant temporal information and the effective fusion of data across multiple modalities. This paper introduces MMFformer, a multimodal depression detection network designed to retrieve depressive spatio-temporal high-level patterns from multimodal social media information. The transformer network with residual connections captures spatial features from videos, and a transformer encoder is exploited to design important temporal dynamics in audio. Moreover, the fusion architecture fused the extracted features through late and intermediate fusion strategies to find out the most relevant intermodal correlations among them. Finally, the proposed network is assessed on two large-scale depression detection datasets, and the results clearly reveal that it surpasses existing state-of-the-art approaches, improving the F1-Score by 13.92% for D-Vlog dataset and 7.74% for LMVD dataset. The code is made available publicly at https://github.com/rezwanh001/Large-Scale-Multimodal-Depression-Detection.