FairX: A comprehensive benchmarking tool for model analysis using fairness, utility, and explainability

TL;DR

FairX is an open-source benchmarking tool that evaluates models on fairness, utility, and explainability, supporting fair generative models and synthetic data evaluation for tabular and image datasets.

Contribution

FairX introduces support for fair generative models and synthetic data evaluation within a unified benchmarking framework, filling gaps in existing tools.

Findings

Supports both tabular and image datasets.

Includes fair generative models and synthetic data evaluation.

Provides comprehensive fairness, utility, and explainability metrics.

Abstract

We present FairX, an open-source Python-based benchmarking tool designed for the comprehensive analysis of models under the umbrella of fairness, utility, and eXplainability (XAI). FairX enables users to train benchmarking bias-mitigation models and evaluate their fairness using a wide array of fairness metrics, data utility metrics, and generate explanations for model predictions, all within a unified framework. Existing benchmarking tools do not have the way to evaluate synthetic data generated from fair generative models, also they do not have the support for training fair generative models either. In FairX, we add fair generative models in the collection of our fair-model library (pre-processing, in-processing, post-processing) and evaluation metrics for evaluating the quality of synthetic fair data. This version of FairX supports both tabular and image datasets. It also allows users to provide their own custom datasets. The open-source FairX benchmarking package is publicly available at \url{https://github.com/fahim-sikder/FairX}.