Assessing Classifier Fairness with Collider Bias

TL;DR

This paper addresses collider bias in machine learning fairness assessments, proposing theorems and an unbiased algorithm that significantly reduce bias, improving the reliability of classifier audits in real-world applications.

Contribution

It introduces theorems to identify and avoid collider bias in fairness evaluation and develops an unbiased assessment algorithm for classifier auditing.

Findings

The proposed algorithm effectively reduces collider bias in fairness assessments.

Experiments demonstrate improved accuracy in fairness evaluation.

The approach is promising for real-world classifier auditing.

Abstract

The increasing application of machine learning techniques in everyday decision-making processes has brought concerns about the fairness of algorithmic decision-making. This paper concerns the problem of collider bias which produces spurious associations in fairness assessment and develops theorems to guide fairness assessment avoiding the collider bias. We consider a real-world application of auditing a trained classifier by an audit agency. We propose an unbiased assessment algorithm by utilising the developed theorems to reduce collider biases in the assessment. Experiments and simulations show the proposed algorithm reduces collider biases significantly in the assessment and is promising in auditing trained classifiers.