MAVias: Mitigate any Visual Bias

TL;DR

MAVias is a novel open-set bias mitigation method that uses foundation models to discover and mitigate diverse visual biases in datasets, improving fairness in computer vision models.

Contribution

The paper introduces MAVias, a new approach leveraging foundation models for discovering and mitigating multiple, unknown visual biases in datasets.

Findings

Effectively detects and mitigates a wide range of biases.

Outperforms current state-of-the-art bias mitigation methods.

Works across diverse datasets like CelebA, Waterbirds, ImageNet, UrbanCars.

Abstract

Mitigating biases in computer vision models is an essential step towards the trustworthiness of artificial intelligence models. Existing bias mitigation methods focus on a small set of predefined biases, limiting their applicability in visual datasets where multiple, possibly unknown biases exist. To address this limitation, we introduce MAVias, an open-set bias mitigation approach leveraging foundation models to discover spurious associations between visual attributes and target classes. MAVias first captures a wide variety of visual features in natural language via a foundation image tagging model, and then leverages a large language model to select those visual features defining the target class, resulting in a set of language-coded potential visual biases. We then translate this set of potential biases into vision-language embeddings and introduce an in-processing bias mitigation approach to prevent the model from encoding information related to them. Our experiments on diverse datasets, including CelebA, Waterbirds, ImageNet, and UrbanCars, show that MAVias effectively detects and mitigates a wide range of biases in visual recognition tasks outperforming current state-of-the-art.