Simulating Biases for Interpretable Fairness in Offline and Online Classifiers

TL;DR

This paper introduces a framework for generating synthetic biased datasets using an agent-based model to evaluate fairness in classifiers, along with a novel explainability technique to assess mitigation effects.

Contribution

It presents a new method for synthetic dataset generation with controllable biases and a novel explainability approach to analyze mitigation impacts on classifiers.

Findings

Synthetic datasets effectively simulate systemic biases.

Mitigation strategies influence how classifiers utilize data features.

The explainability technique reveals the impact of bias mitigation methods.

Abstract

Predictive models often reinforce biases which were originally embedded in their training data, through skewed decisions. In such cases, mitigation methods are critical to ensure that, regardless of the prevailing disparities, model outcomes are adjusted to be fair. To assess this, datasets could be systematically generated with specific biases, to train machine learning classifiers. Then, predictive outcomes could aid in the understanding of this bias embedding process. Hence, an agent-based model (ABM), depicting a loan application process that represents various systemic biases across two demographic groups, was developed to produce synthetic datasets. Then, by applying classifiers trained on them to predict loan outcomes, we can assess how biased data leads to unfairness. This highlights a main contribution of this work: a framework for synthetic dataset generation with controllable bias injection. We also contribute with a novel explainability technique, which shows how mitigations affect the way classifiers leverage data features, via second-order Shapley values. In experiments, both offline and online learning approaches are employed. Mitigations are applied at different stages of the modelling pipeline, such as during pre-processing and in-processing.