VB-Mitigator: An Open-source Framework for Evaluating and Advancing Visual Bias Mitigation

TL;DR

VB-Mitigator is an open-source framework that standardizes evaluation and comparison of visual bias mitigation techniques in computer vision, promoting reproducibility and accelerating progress toward fairer AI systems.

Contribution

It introduces a unified, extensible platform with multiple mitigation methods, datasets, and evaluation practices to advance research in visual bias mitigation.

Findings

Provides a comprehensive benchmark of state-of-the-art methods.

Facilitates reproducibility and fair comparison across studies.

Supports integration of new methods and datasets easily.

Abstract

Bias in computer vision models remains a significant challenge, often resulting in unfair, unreliable, and non-generalizable AI systems. Although research into bias mitigation has intensified, progress continues to be hindered by fragmented implementations and inconsistent evaluation practices. Disparate datasets and metrics used across studies complicate reproducibility, making it difficult to fairly assess and compare the effectiveness of various approaches. To overcome these limitations, we introduce the Visual Bias Mitigator (VB-Mitigator), an open-source framework designed to streamline the development, evaluation, and comparative analysis of visual bias mitigation techniques. VB-Mitigator offers a unified research environment encompassing 12 established mitigation methods, 7 diverse benchmark datasets. A key strength of VB-Mitigator is its extensibility, allowing for seamless integration of additional methods, datasets, metrics, and models. VB-Mitigator aims to accelerate research toward fairness-aware computer vision models by serving as a foundational codebase for the research community to develop and assess their approaches. To this end, we also recommend best evaluation practices and provide a comprehensive performance comparison among state-of-the-art methodologies.