Fair-MoE: Fairness-Oriented Mixture of Experts in Vision-Language Models

TL;DR

Fair-MoE is a novel vision-language model designed to enhance fairness and accuracy in medical applications by leveraging a specialized mixture of experts and a fairness-oriented loss function, addressing bias concerns in medical AI.

Contribution

It introduces Fair-MoE, combining a fairness-oriented mixture of experts and a new loss function to improve fairness and effectiveness in medical vision-language models.

Findings

Improves fairness across multiple attributes.

Enhances accuracy in medical VLM tasks.

Demonstrates effectiveness on Harvard-FairVLMed dataset.

Abstract

Fairness is a fundamental principle in medical ethics. Vision Language Models (VLMs) have shown significant potential in the medical field due to their ability to leverage both visual and linguistic contexts, reducing the need for large datasets and enabling the performance of complex tasks. However, the exploration of fairness within VLM applications remains limited. Applying VLMs without a comprehensive analysis of fairness could lead to concerns about equal treatment opportunities and diminish public trust in medical deep learning models. To build trust in medical VLMs, we propose Fair-MoE, a model specifically designed to ensure both fairness and effectiveness. Fair-MoE comprises two key components: \textit{the Fairness-Oriented Mixture of Experts (FO-MoE)} and \textit{the Fairness-Oriented Loss (FOL)}. FO-MoE is designed to leverage the expertise of various specialists to filter out biased patch embeddings and use an ensemble approach to extract more equitable information relevant to specific tasks. FOL is a novel fairness-oriented loss function that not only minimizes the distances between different attributes but also optimizes the differences in the dispersion of various attributes' distributions. Extended experiments demonstrate the effectiveness and fairness of Fair-MoE. Tested on the Harvard-FairVLMed dataset, Fair-MoE showed improvements in both fairness and accuracy across all four attributes. Code will be publicly available.