Reading Between the Frames: Multi-Modal Depression Detection in Videos from Non-Verbal Cues

TL;DR

This paper introduces a multi-modal temporal model that effectively detects depression from user-generated videos by analyzing diverse non-verbal cues, achieving state-of-the-art results on benchmark datasets.

Contribution

The work presents a novel flexible multi-modal model that leverages high-level non-verbal cues from videos for depression detection, filling a gap in video-based mental health assessment.

Findings

Achieves state-of-the-art performance on three benchmark datasets.

Utilizes diverse non-verbal cues including facial, bodily, and gaze features.

Demonstrates the importance of high-level cues in real-world, noisy videos.

Abstract

Depression, a prominent contributor to global disability, affects a substantial portion of the population. Efforts to detect depression from social media texts have been prevalent, yet only a few works explored depression detection from user-generated video content. In this work, we address this research gap by proposing a simple and flexible multi-modal temporal model capable of discerning non-verbal depression cues from diverse modalities in noisy, real-world videos. We show that, for in-the-wild videos, using additional high-level non-verbal cues is crucial to achieving good performance, and we extracted and processed audio speech embeddings, face emotion embeddings, face, body and hand landmarks, and gaze and blinking information. Through extensive experiments, we show that our model achieves state-of-the-art results on three key benchmark datasets for depression detection from video by a substantial margin. Our code is publicly available on GitHub.