Exploring Machine Learning and Language Models for Multimodal Depression Detection

TL;DR

This paper evaluates various machine learning and deep learning models, including XGBoost, transformers, and LLMs, for multimodal depression detection across audio, video, and text data, providing insights into effective strategies for mental health prediction.

Contribution

It introduces a comprehensive comparison of models for multimodal depression detection, highlighting their strengths and limitations across different data modalities.

Findings

Transformers outperform XGBoost in certain modalities.

LLMs show promise in text-based depression detection.

Multimodal approaches improve detection accuracy.

Abstract

This paper presents our approach to the first Multimodal Personality-Aware Depression Detection Challenge, focusing on multimodal depression detection using machine learning and deep learning models. We explore and compare the performance of XGBoost, transformer-based architectures, and large language models (LLMs) on audio, video, and text features. Our results highlight the strengths and limitations of each type of model in capturing depression-related signals across modalities, offering insights into effective multimodal representation strategies for mental health prediction.

Tables

Table 1. Table 3: Ablation study on each systems for 5s MPDD Elderly binary dev set (90(cross-validation)/10).

Methods	$W_{F1}$	$U_{F1}$
XGBoost
Raw Feature	65.76	42.62
Raw Feature + class weighting	74.80	65.04
PCA Feature + class weighting	94.29	91.90
Transformer
2 transformer layers	80.44	69.64
2 transformer layers + mixup	84.00	76.22
2 transformer layers +mixup + cross-validation	87.08	83.19
LLM
Llama2	60.96	44.57
EmotionLlama2 + 1step	31.77	33.33
EmotionLlama2 + 2steps	70.59	52.55

Table 2. Table 4: Weighted F1 and Unweighted F1 (%) ↑ \uparrow results on MPDD-Elderly dev set.

Method	PF	1s						5s
		Binary		Ternary		Quinary		Binary		Ternary		Quinary
		$W_{F1}$	$U_{F1}$	$W_{F1}$	$U_{F1}$	$W_{F1}$	$U_{F1}$	$W_{F1}$	$U_{F1}$	$W_{F1}$	$U_{F1}$	$W_{F1}$	$U_{F1}$
Baseline	✗	82.60	70.89	54.35	49.14	63.85	44.00	77.90	66.15	50.88	47.59	73.49	56.83
Baseline	✓	85.71	79.13	56.48	55.64	66.26	46.66	81.75	72.37	58.22	59.37	75.62	58.40
XGBoost	✗	90.67	85.83	55.23	53.02	55.60	21.43	94.29	91.90	61.02	62.51	54.62	21.05
Transformer	✓	93.44	88.21	74.95	80.00	82.21	46.77	85.27	71.55	65.52	61.31	67.96	67.62
LLM	✓	70.59	52.55	61.17	53.02	77.89	30.60	67.43	45.60	45.66	37.44	77.89	30.60

Table 3. Table 5: Weighted F1 and Unweighted F1 (%) ↑ \uparrow results on MPDD-Young dev set with model size for the binary task.

Method	PF	#Params(M)	1s				5s
			Binary		Ternary		Binary		Ternary
			$W_{F1}$	$U_{F1}$	$W_{F1}$	$U_{F1}$	$W_{F1}$	$U_{F1}$	$W_{F1}$	$U_{F1}$
Baseline	✗	1.89111The parameters of the baseline system are calculated using wav2vec and OpenFace features as the input features.	55.23	55.23	47.95	43.72	60.02	60.02	42.82	39.38
Baseline	✓	2.15	59.96	59.96	51.86	51.62	62.11	62.11	48.18	41.31
XGBoost	✓	0.002	81.53	81.38	66.67	48.89	74.07	74.07	62.19	45.60
Transformer	✓	1.06	95.83	95.83	75.60	71.36	81.48	81.48	78.51	59.16
LLM	✓	6,843	60.86	61.65	45.65	34.06	64.07	64.96	39.49	28.82

Equations

$$Z_a^{(i)}$$

$$Z_v^{(i)}$$

$$z_t^{(i)}$$

$${\hat{Z}}_a^{(i)}=Z_a^{(i)}+P_a,{\hat{Z}}_v^{(i)}=Z_v^{(i)}+P_v.$$

$$H_a^{(i)}$$

$$H_v^{(i)}$$

$${\tilde{x}}_a^{(i)}$$

$${\tilde{x}}_v^{(i)}$$

$$O=\phi\big{(}\sigma_{\text{aud}}(X_{a}^{(i)}),\sigma_{\text{vis}}(X_{v}^{(i)}),\sigma_{\text{txt}}(x_{t}^{(i)}),\text{Tokenizer}(\text{Prompt})\big{)}.$$

Full text

1 Introduction

The World Health Organization recently reported that depression affects 3.8% of the global population and 63.6% of these cases remain undiagnosed [1], partly due to the limited availability of healthcare services and, in some cases, financial constraints that prevent many individuals from accessing necessary medical care [2]. Traditional methods, which rely primarily on self-reported questionnaires such as the PHQ-9 [3] and the BDI-II [4], are limited in their ability to capture the dynamic and multifaceted nature of depressive symptoms. These methods are also prone to reporting biases and may fail to detect early or subtle changes in depressive states.

In response to these challenges, the computing community has been instrumental in advancing automatic depression detection by leveraging multimodal data [5, 6, 7, 8, 9, 10, 11, 12, 13]. Most recently, the first Multimodal Personality-Aware Depression Detection (MPDD) Challenge [14] introduced a richly annotated novel dataset that includes audio and visual recordings of participants engaging in a variety of real-world scenarios. The MPDD dataset is annotated using the PHQ-9 scale, Big Five personality traits [15], and detailed demographic information. Compared to existing corpora, the MPDD dataset offers greater contextual diversity and annotation depth, enabling more inclusive and fine-grained modeling.

In terms of methodological approaches, current research increasingly integrates audio, visual, and textual biomarkers through progressively sophisticated computational paradigms. Given that depression detection is a form of fine-grained, subtle emotion recognition, many researchers have drawn inspiration from emotion classification methods. Traditional machine learning approaches typically rely on handcrafted feature extraction pipelines. For example, OpenSMILE-derived acoustic features [16] are often combined with facial expression metrics extracted using computer vision tools, such as Action Units from OpenFace [17] or emotion probabilities from facial emotion recognition models [18]. These features are typically fed into ensemble classifiers, such as XGBoost [19], or kernel-based methods, like support vector machine [20]. Some studies further enhance performance using feature selection or principal component analysis (PCA)-based dimensionality reduction prior to classification [21].

More advanced systems employ hybrid architectures with modality-specific processing pipelines. A common configuration involves using separate convolutional neural networks (CNNs) for visual frames, long short-term memory (LSTM) networks for audio spectrograms, and transformer networks for textual inputs [22]. Fusion strategies vary, ranging from early fusion of low-level features to late fusion of modality-specific predictions [23]. Recent work has also explored cross-modal attention mechanisms using transformers to learn joint representations [24]. Further progress has been achieved through unified architectures that combine self-supervised audio encoders for paralinguistic feature extraction, vision transformers for modeling spatio-temporal facial dynamics, and specialized language models for clinical text analysis [25]. In recent years, large language models (LLMs) have begun to reshape the field. Multimodal emotion recognition systems such as Emotion-LLaMA [26] integrate audio, visual, and textual inputs through emotion-specific encoders. By aligning these features within a shared latent space and applying a modified LLaMA model with instruction tuning, Emotion-LLaMA significantly enhances both emotional recognition and reasoning capabilities.

Among these approaches, XGBoost, transformer-based models, and LLMs have each achieved state-of-the-art results at different stages of research. However, the MPDD Challenge launched in 2025 introduces new complexities that may affect model performance when applying different methods. Inspired by this, the present study systematically evaluates and compares the effectiveness of three representative model classes, XGBoost, transformer-based models, and LLMs, on the MPDD dataset. Following the official challenge protocol, we assess the strengths and limitations of each model across modalities, with the goal of identifying their respective potentials for advancing real-world depression recognition systems.

2 The first Multimodal Personality-aware Depression Detection Challenge

This section provides an overview of the MPDD setup, including the datasets, available audio, visual, and text features, as well as the baseline system, which will serve as the foundation for this study.

2.1 Datasets

The MPDD dataset comprises two tracks corresponding to distinct age groups, MPDD-Elderly and MPDD-Young, designed to facilitate age-specific depression analysis. Table 1 provides a comprehensive summary of class distributions across three classification tasks (binary, trinary, and quinary) for both subsets, reporting sample counts, class ratios, and the number of unique speakers (patients)111Note that the table only lists the statistics of the MPDD training set. As the time we are writing, the labels of test set are not available. In the following experiments, we split the training set into a 90-10 ratio, using 10% as the development set and reporting the results based on this split..

2.1.1 Track 1: MPDD-Elderly

Track 1 focuses on depression detection among elderly participants (average age: 62.8 $\pm$ 11.0). Data were collected through semi-structured interviews conducted in hospital settings. Each participant completed standardized clinical questionnaires, including the PHQ-9 and HAMD-24 scales [27], to assess depression severity. HAMD-24 scores are used to generate labels for the binary and ternary classification tasks, while PHQ-9 scores are used for the quinary classification task.

To enable a more comprehensive participant profile, additional annotations are provided, including Big five personality traits (using a 10-point scale) [15], physical health conditions, financial stress levels, and the number of cohabiting family members, see Table 2.

2.1.2 Track 2: MPDD-Young

Track 2 targets a younger population (average age: 20.0 $\pm$ 2.2), recruited in non-clinical environments. The data collection protocol consists of a self-introduction, a questionnaire segment, and a scripted reading task, all recorded via video. Participants completed the PHQ-9 questionnaire, which is used to generate depression labels for the binary and ternary classification tasks.

Personality trait annotations in this track differ slightly from those in MPDD-Elderly. In addition to the Big five traits, demographic variables such as age, gender, and place of origin are included, allowing for comparative analysis across different population groups, see Table 2.

2.2 Feature Modalities

Audio features include Mel-frequency cepstral coefficients (MFCCs), low-level acoustic descriptors extracted using OpenSMILE [16], and deep learning-based representations from pre-trained models such as Wav2Vec 2.0 [28]. These features provide a comprehensive view of both handcrafted and learned paralinguistic cues. Visual features comprise deep CNN-based facial embeddings obtained using architectures like DenseNet and ResNet [29, 30], as well as facial behavior analysis (e.g., eye gaze, and head pose) extracted using OpenFace [17]. For the textual modality, both tracks offer RoBERTa-based embeddings [31] derived from raw personality traits descriptions. Each feature type is provided as a fixed-length embedding per 1-second or 5-second hopping window. These variable-length sequences are temporally aligned on a per-subject basis to maintain consistency across modalities.

2.3 Baseline System

The official baseline system adopts a multimodal deep learning approach, as illustrated on the left of Figure 1. Audio and visual features are first passed through modality-specific encoders, implemented as one-layer LSTM. An optional personalized feature, extracted from a RoBERTa-large model, can be concatenated with the LSTM-processed audio and visual embeddings. The fused representation is then passed through a one-layer transformer that integrates the multimodal information. Finally, the output is fed into fully connected layers to predict the number of classes corresponding to depression severity levels. The system is trained end-to-end using a combination of cross-entropy loss and focal loss [32], and is evaluated using weighted/unweighted F1 scores and overall accuracy.

3 Investigated Systems

This section elaborates on the systems we explored for multimodal depression detection, covering traditional machine learning models, deep learning models, and LLMs.

3.1 XGBoost-Based Model

We implemented a gradient-boosted decision tree pipeline using XGBoost for multimodal depression classification based on audio and visual features222We attempted to incorporate personalized features but did not observe performance improvements; thus, we excluded them from the XGBoost model., as illustrated in the second panel of Figure 1. Each input sample consists of fixed-length, pre-extracted embeddings derived from pretrained models and represent 1s or 5s window-level summaries of audio and video segments, denoted as $X_{a}\in\mathbb{R}^{n\times d_{a}}$ and $X_{v}\in\mathbb{R}^{n\times d_{v}}$, respectively, where $n$ is the number of frames for the audio and visual streams, respectively, and $d_{a}$, $d_{v}$ are the corresponding feature dimensions.

To reduce redundancy and improve generalization, we applied Principal Component Analysis (PCA) separately to each modality. For each $X_{m}\in\{X_{a},X_{v}\}$, we centered the data, computed the covariance matrix $C_{m}$, and extracted the top-$k$ eigenvectors $V_{m}^{k}$. Each modality was then projected as $Z_{m}=(X_{m}-\text{mean}(X_{m}))V_{m}^{k}\in\mathbb{R}^{n\times k}$, with $k=50$. The reduced audio and visual features $Z_{a}$ and $Z_{v}$ were concatenated to form a fused multimodal embedding $Z=[Z_{a}\|Z_{v}]\in\mathbb{R}^{n\times 100}$, which served as input to the XGBoost classifier.

XGBoost is trained for $T$ boosting rounds. At each round $t$, the model computes the gradients $g_{t}$ and Hessians $h_{t}$ of the multi-class log loss with respect to the current predictions $\hat{y}^{(t-1)}$, fits a regression tree $f_{t}$ to predict the gradients, and updates the predictions as: $\hat{y}^{(t)}\leftarrow\hat{y}^{(t-1)}+f_{t}(Z)$. The final model aggregates all trees: $\hat{y}(x)=\sum_{t=1}^{T}f_{t}(x)$.

Besides XGBoost, to address class imbalance, we apply class weighting by assigning a higher weight to the minority class. Specifically, the positive class weight is computed as: $w_{\text{pos}}=\frac{N_{\text{neg}}}{N_{\text{pos}}}$, where $N_{\text{neg}}$ and $N_{\text{pos}}$ are the number of negative and positive samples, respectively. For example, with 90 negatives and 10 positives, $w_{\text{pos}}=9$. This helps the model better learn from underrepresented classes.

3.2 Transformer-Based Model

We design a multimodal transformer model to fuse audio, visual, and text features for depression classification.

Each input sample $i$ contains audio $X_{a}^{(i)}\in\mathbb{R}^{T_{a}\times d_{a}}$, visual $X_{v}^{(i)}\in\mathbb{R}^{T_{v}\times d_{v}}$, and text features $x_{t}^{(i)}\in\mathbb{R}^{d_{t}}$. These inputs are projected into a shared latent space of dimension $d$ using learned linear layers followed by layer normalization:

[TABLE]

where $W_{a}\in\mathbb{R}^{d_{a}\times d}$, $W_{v}\in\mathbb{R}^{d_{v}\times d}$, and $W_{t}\in\mathbb{R}^{d_{t}\times d}$ are learnable projection matrices. Positional encodings $P_{a}\in\mathbb{R}^{T_{a}\times d}$ and $P_{v}\in\mathbb{R}^{T_{v}\times d}$ are added to preserve temporal order:

[TABLE]

The temporally encoded sequences are passed through modality-specific transformer encoders:

[TABLE]

To obtain fixed-length representations, we apply learned attention pooling over time:

[TABLE]

The pooled audio, pooled visual, and text features are concatenated to form the multimodal representation: $z^{(i)}=\left[\tilde{x}_{a}^{(i)}\;\|\;\tilde{x}_{v}^{(i)}\;\|\;z_{t}^{(i)}\right]\in\mathbb{R}^{3d}.$ Finally, $z^{(i)}$ is passed through two fully connected layers with ReLU activation, layer normalization, and dropout to predict depression severity. The focal loss function is used to handle class imbalance.

It is well recognized that deep learning models are prone to overfitting when trained on small datasets, as is the case with the MPDD dataset. To address this, we apply Mixup data augmentation [33] during training, which has been shown to improve model robustness and performance across various deep learning tasks. The mixup augmentation strategy creates synthetic training examples by linearly interpolating between pairs of samples and their corresponding labels. Specifically, a new example is generated by combining the two input samples and labels using a mixing coefficient.

3.3 LLM-Based Model

We also explore the use of LLM-based methods for the MPDD task. Specifically, we are inspired by Emotion-LLaMA [26], a model built upon the LLaMA backbone and fine-tuned on large-scale multimodal emotion datasets. Emotion-LLaMA enables emotion reasoning by integrating visual, auditory, and textual cues through structured prompts and cross-modal attention.

Building upon this foundation, we adapt Emotion-LLaMA to the MPDD task by fine-tuning it on our multimodal dataset. The model formulation is expressed as:

[TABLE]

Here, the input consists of audio features $X_{a}^{(i)}$, visual features $X_{v}^{(i)}$, textual features $x_{t}^{(i)}$, and a task-specific prompt in a multiple-choice question format (as illustrated on the rightmost side of Figure 1).

To integrate features from multiple modalities, we introduce a linear projection mechanism that maps each modality into a shared embedding space. This is achieved via trainable linear projection functions: $\sigma_{\text{aud}}$ for audio, $\sigma_{\text{vis}}$ for visual, and $\sigma_{\text{txt}}$ for text. The final output $O$ is a formatted text response, also shown in the rightmost bottom of Figure 1.

The fine-tuning process involves two stages. In Stage 1, the LLaMA backbone is frozen, and only the projection layers and classification heads are trained. In Stage 2, LoRA-based fine-tuning is applied, using a dual learning rate setup to update both the LoRA parameters and the projection layers.

4 Experiments

Experiments begin with the 5-second MPDD-Elderly dataset, where we conduct a comprehensive ablation study to identify the optimal configuration for each system. Once the best settings are determined, these configurations are applied to the remaining scenarios. For all scenarios, we use a 90-10 train-validation split based on patient IDs, ensuring subject independence and preventing information leakage. We evaluated each configuration using two primary metrics: weighted ($W_{F1}$) and unweighted F1 scores ($U_{F1}$).

4.1 Experiment Settings

4.1.1 XGBoost Setting

The XGBoost configuration was carefully designed to balance interpretability, computational efficiency, and robust performance on limited multimodal data. Each audio and visual modality was first reduced to 50 dimensions using PCA, resulting in a fused 100-dimensional feature vector. For configurations without PCA, original modality features were concatenated directly. The model used a shallow tree depth with a maximum depth of 3, a learning rate selected from $\{0.01,0.05\}$, and both subsample and colsample-by-tree ratios set to 0.8 to introduce randomness and reduce overfitting. Training employed up to 500 boosting rounds with early stopping after 25 rounds without improvement, using multi-class log loss as the evaluation metric and multi:softprob as the objective function. Hyperparameters were tuned per modality pair and classification task (binary, ternary, quinary) using early stopping on a speaker-level validation split to avoid overfitting and data leakage. Multiple audio-visual fusion pairs were explored (e.g., MFCC + OpenFace, OpenSMILE + ResNet), with MFCC + OpenFace achieving the best results. Consistency was maintained by applying the same patient-level ID split and evaluation methodology across all experiments. Notably, personality-aware features and probabilistic model ensembling were excluded to enable a focused evaluation of core modality fusion performance under classical machine learning settings.

4.1.2 Transformer setting

The transformer configuration prioritizes efficiency and regularization to avoid overfitting on the relatively small dataset. The model uses a reduced dimensionality of 128, with a shallow 2-layer transformer architecture and 4 attention heads, which together provide sufficient representational capacity while maintaining computational efficiency. A dropout rate of 0.5 is applied throughout the network for strong regularization. The pre-classifier network reduces the concatenated multimodal embedding from 512 to 256 dimensions, refining the joint feature representation for downstream classification. For training, we use a learning rate of 5e-5, batch size of 2, and train for up to 100 epochs, with early stopping triggered if no improvement is observed for 20 consecutive epochs. We also include warmup training for the first 10 epochs, gradient clipping at 1.0, and apply weight decay of 1e-4 to further aid generalization. Multiple audio-visual fusion pairs were explored, and wav2vec2, DenseNet, with personalized features were selected.

For the mixup augmentation strategy, the mixing coefficient is sampled from a Beta distribution with parameters 0.2 and 0.2. Applied consistently across all modalities with a 50% probability, mixup helps the model learn smoother decision boundaries and improves generalization on limited data. The cross-validation experiment employs 10-fold validation, where each fold naturally provides the 90-10 split used in strategies without cross-validation.

4.1.3 LLM setting

The Depression-LLaMA implementation is based on the Emotion-LLaMA pre-trained foundation, utilizing the LLaMA-2-7B333https://huggingface.co/meta-llama/Llama-2-7b as the underlying large language model. The model architecture utilize all features, including three audio features, three visual feature and one text features listed in table 2 Each of them will be mapped to 4,096 dimensional features by the linear projection. The training process is conducted in two stages. In Stage 1, the backbone is frozen and trained for 5 epochs with a learning rate of $5\times 10^{-5}$. In Stage 2, LoRA fine-tuning is applied for 3 epochs with a learning rate of $1\times 10^{-5}$.

4.2 Results

4.2.1 Ablation Study for each system

Table 3 presents the weighted F1 ($W_{F1}$) and unweighted F1 ($U_{F1}$) scores for various systems evaluated on the 5-second MPDD Elderly binary development set using a 90-10 split (with cross-validation where specified). The results compare baseline methods, traditional machine learning approaches, transformer models, and LLM baselines, with different training strategies.

XGBoost: Starting with raw features, XGBoost achieves moderate performance. Incorporating class weighting leads to a notable improvement, and applying PCA alongside class weighting further enhances results, demonstrating the effectiveness of dimensionality reduction and handling class imbalance.

Transformer: The baseline transformer outperforms XGBoost without PCA, showing strong capability in modeling the data. Adding mixup augmentation further improves the model’s generalization, and combining mixup with cross-validation yields the best and most robust performance, highlighting the benefits of data augmentation and rigorous evaluation.

LLM Methods: The baseline LLaMA2 model performs relatively poorly compared to other methods. Initial fine-tuning of EmotionLLaMA2 results in a performance drop, likely due to adaptation challenges, but subsequent fine-tuning improves outcomes considerably. Despite this, LLM-based methods still lag behind the transformer and XGBoost models on this task.

4.2.2 Overall Results

Table 4 and Table 5 present a comprehensive comparison of all explored methods on the MPDD-Elderly dataset. On the MPDD-Elderly development set, XGBoost achieves the highest weighted and unweighted F1 scores for the 5-second binary classification task, demonstrating strong performance on longer audio segments with effective feature engineering. The Transformer model outperforms XGBoost on shorter 1-second segments and more complex classification tasks (ternary and quinary), indicating its strength in capturing fine-grained information. Personalized features improve baseline models but the LLM-based approach shows comparatively lower performance across most tasks.

For the MPDD-Young development set, the Transformer consistently delivers the best results across all classification tasks and time windows, with weighted F1 scores reaching as high as 95.83% on 1-second binary classification. XGBoost performs well but lags behind Transformer models, especially on ternary tasks. Baseline models benefit from personalized features but remain less competitive, while LLM-based methods perform the weakest.

Overall, the Transformer model (1.06M parameters) is the most effective for multimodal depression detection, especially for younger speakers and shorter audio windows. Despite having significantly more parameters, LLM (6,843M) underperforms, especially in ternary classification, suggesting that larger models don’t always guarantee better performance. XGBoost, with only 0.002M parameters, achieves strong results in binary classification, highlighting the effectiveness of simpler models for specific tasks. The Baseline with a personalized feature slightly improves on the baseline without it but remains less effective than more complex models, even though it has more parameters (2.15M) than the Transformer.

Acknowledgment This research is supported by the Ministry of Education, Singapore, under its Academic Research Tier 1 (Grant number: GMS 956) and the Academy of Medical Sciences, under its Networking Grant (NGR1\1678).