Revisiting Pre-processing Group Fairness: A Modular Benchmarking Framework

TL;DR

This paper introduces FairPrep, a modular benchmarking framework built on AIF360, to evaluate pre-processing fairness methods on tabular data, addressing a gap in standardized evaluation tools for data-level fairness.

Contribution

The paper presents FairPrep, a flexible, extensible benchmarking framework for pre-processing fairness techniques, enabling standardized, reproducible evaluation on tabular datasets.

Findings

FairPrep facilitates comprehensive fairness evaluation.

It supports seamless integration of datasets, interventions, and models.

The framework promotes reproducibility and standardization in fairness research.

Abstract

As machine learning systems become increasingly integrated into high-stakes decision-making processes, ensuring fairness in algorithmic outcomes has become a critical concern. Methods to mitigate bias typically fall into three categories: pre-processing, in-processing, and post-processing. While significant attention has been devoted to the latter two, pre-processing methods, which operate at the data level and offer advantages such as model-agnosticism and improved privacy compliance, have received comparatively less focus and lack standardised evaluation tools. In this work, we introduce FairPrep, an extensible and modular benchmarking framework designed to evaluate fairness-aware pre-processing techniques on tabular datasets. Built on the AIF360 platform, FairPrep allows seamless integration of datasets, fairness interventions, and predictive models. It features a batch-processing interface that enables efficient experimentation and automatic reporting of fairness and utility metrics. By offering standardised pipelines and supporting reproducible evaluations, FairPrep fills a critical gap in the fairness benchmarking landscape and provides a practical foundation for advancing data-level fairness research.