FAIR_XAI: Improving Multimodal Foundation Model Fairness via Explainability for Wellbeing Assessment

TL;DR

This paper investigates the fairness and explainability of Vision-Language Models in wellbeing assessment, revealing performance variability, biases, and the complex effects of explainability interventions on fairness.

Contribution

It introduces an explainability framework for VLMs in wellbeing assessment, analyzing biases and fairness trade-offs across datasets and architectures.

Findings

Phi3.5-Vision achieved 80.4% accuracy on E-DAIC.

Qwen2-VL struggled at 33.9% accuracy and showed higher gender bias.

Explainability interventions had mixed effects on fairness and bias.

Abstract

In recent years, the integration of multimodal machine learning in wellbeing assessment has offered transformative potential for monitoring mental health. However, with the rapid advancement of Vision-Language Models (VLMs), their deployment in clinical settings has raised concerns due to their lack of transparency and potential for bias. While previous research has explored the intersection of fairness and Explainable AI (XAI), its application to VLMs for wellbeing assessment and depression prediction remains under-explored. This work investigates VLM performance across laboratory (AFAR-BSFT) and naturalistic (E-DAIC) datasets, focusing on diagnostic reliability and demographic fairness. Performance varied substantially across environments and architectures; Phi3.5-Vision achieved 80.4% accuracy on E-DAIC, while Qwen2-VL struggled at 33.9%. Additionally, both models demonstrated a tendency to over-predict depression on AFAR-BSFT. Although bias existed across both architectures, Qwen2-VL showed higher gender disparities, while Phi-3.5-Vision exhibited more racial bias. Our XAI intervention framework yielded mixed results; fairness prompting achieved perfect equal opportunity for Qwen2-VL at a severe accuracy cost on E-DAIC. On AFAR-BSFT, explainability-based interventions improved procedural consistency but did not guarantee outcome fairness, sometimes amplifying racial bias. These results highlight a persistent gap between procedural transparency and equitable outcomes. We analyse these findings and consolidate concrete recommendations for addressing them, emphasising that future fairness interventions must jointly optimise predictive accuracy, demographic parity, and cross-domain generalisation.