Tri-Modal Severity Fused Diagnosis across Depression and Post-traumatic Stress Disorders

TL;DR

This paper introduces a tri-modal affective severity framework that fuses text, audio, and facial signals to diagnose depression and PTSD, providing more nuanced, robust, and explainable assessments than unimodal methods.

Contribution

It presents a novel multi-modal fusion approach for severity estimation across depression and PTSD, improving robustness and interpretability over existing unimodal models.

Findings

Outperforms unimodal and ablation baselines in cross-validation.

Improves decision curve utility and robustness under noisy or missing data.

Reduces regression error and enhances class concordance for PTSD severity.

Abstract

Depression and post traumatic stress disorder (PTSD) often co-occur with connected symptoms, complicating automated assessment, which is often binary and disorder specific. Clinically useful diagnosis needs severity aware cross disorder estimates and decision support explanations. Our unified tri modal affective severity framework synchronizes and fuses interview text with sentence level transformer embeddings, audio with log Mel statistics with deltas, and facial signals with action units, gaze, head and pose descriptors to output graded severities for diagnosing both depression (PHQ-8; 5 classes) and PTSD (3 classes). Standardized features are fused via a calibrated late fusion classifier, yielding per disorder probabilities and feature-level attributions. This severity aware tri-modal affective fusion approach is demoed on multi disorder concurrent depression and PTSD assessment. Stratified cross validation on DAIC derived corpora outperforms unimodal/ablation baselines. The fused model matches the strongest unimodal baseline on accuracy and weighted F1, while improving decision curve utility and robustness under noisy or missing modalities. For PTSD specifically, fusion reduces regression error and improves class concordance. Errors cluster between adjacent severities; extreme classes are identified reliably. Ablations show text contributes most to depression severity, audio and facial cues are critical for PTSD, whereas attributions align with linguistic and behavioral markers. Our approach offers reproducible evaluation and clinician in the loop support for affective clinical decision making.