FAIRWELL: Fair Multimodal Self-Supervised Learning for Wellbeing Prediction

TL;DR

FAIRWELL introduces a novel multimodal self-supervised learning method that enhances fairness in healthcare prediction tasks by learning subject-independent representations, effectively balancing fairness and accuracy.

Contribution

The paper proposes a new subject-level loss function, FAIRWELL, that adapts VICReg for fair multimodal learning, addressing fairness in healthcare predictions.

Findings

Improves fairness with minimal impact on accuracy.

Enhances the fairness-performance Pareto frontier.

Effective across diverse healthcare datasets.

Abstract

Early efforts on leveraging self-supervised learning (SSL) to improve machine learning (ML) fairness has proven promising. However, such an approach has yet to be explored within a multimodal context. Prior work has shown that, within a multimodal setting, different modalities contain modality-unique information that can complement information of other modalities. Leveraging on this, we propose a novel subject-level loss function to learn fairer representations via the following three mechanisms, adapting the variance-invariance-covariance regularization (VICReg) method: (i) the variance term, which reduces reliance on the protected attribute as a trivial solution; (ii) the invariance term, which ensures consistent predictions for similar individuals; and (iii) the covariance term, which minimizes correlational dependence on the protected attribute. Consequently, our loss function, coined as FAIRWELL, aims to obtain subject-independent representations, enforcing fairness in multimodal prediction tasks. We evaluate our method on three challenging real-world heterogeneous healthcare datasets (i.e. D-Vlog, MIMIC and MODMA) which contain different modalities of varying length and different prediction tasks. Our findings indicate that our framework improves overall fairness performance with minimal reduction in classification performance and significantly improves on the performance-fairness Pareto frontier.