MDD-Net: Multimodal Depression Detection through Mutual Transformer

TL;DR

This paper introduces MDD-Net, a multimodal neural network utilizing mutual transformers to improve depression detection accuracy from social media acoustic and visual data, outperforming previous methods by up to 17.37% in F1-Score.

Contribution

The paper presents a novel multimodal depression detection network that effectively fuses acoustic and visual features using mutual transformers, advancing the state-of-the-art performance.

Findings

Outperforms state-of-the-art by up to 17.37% F1-Score

Efficient multimodal feature fusion with mutual transformers

Validated on the D-Vlog dataset

Abstract

Depression is a major mental health condition that severely impacts the emotional and physical well-being of individuals. The simple nature of data collection from social media platforms has attracted significant interest in properly utilizing this information for mental health research. A Multimodal Depression Detection Network (MDD-Net), utilizing acoustic and visual data obtained from social media networks, is proposed in this work where mutual transformers are exploited to efficiently extract and fuse multimodal features for efficient depression detection. The MDD-Net consists of four core modules: an acoustic feature extraction module for retrieving relevant acoustic attributes, a visual feature extraction module for extracting significant high-level patterns, a mutual transformer for computing the correlations among the generated features and fusing these features from multiple modalities, and a detection layer for detecting depression using the fused feature representations. The extensive experiments are performed using the multimodal D-Vlog dataset, and the findings reveal that the developed multimodal depression detection network surpasses the state-of-the-art by up to 17.37% for F1-Score, demonstrating the greater performance of the proposed system. The source code is accessible at https://github.com/rezwanh001/Multimodal-Depression-Detection.