A Comprehensive Evaluation of the Sensitivity of Density-Ratio Estimation Based Fairness Measurement in Regression

TL;DR

This paper evaluates how different density-ratio estimation methods impact fairness measurement in regression, revealing significant variability and inconsistency in results depending on the core used.

Contribution

It introduces multiple fairness measurement methods with various density-ratio cores and systematically analyzes their sensitivity, highlighting reliability issues in current approaches.

Findings

Choice of density-ratio core significantly affects fairness measurement outcomes.

Different cores can produce inconsistent fairness rankings of algorithms.

Current density-ratio based fairness measures may lack reliability.

Abstract

The prevalence of algorithmic bias in Machine Learning (ML)-driven approaches has inspired growing research on measuring and mitigating bias in the ML domain. Accordingly, prior research studied how to measure fairness in regression which is a complex problem. In particular, recent research proposed to formulate it as a density-ratio estimation problem and relied on a Logistic Regression-driven probabilistic classifier-based approach to solve it. However, there are several other methods to estimate a density ratio, and to the best of our knowledge, prior work did not study the sensitivity of such fairness measurement methods to the choice of underlying density ratio estimation algorithm. To fill this gap, this paper develops a set of fairness measurement methods with various density-ratio estimation cores and thoroughly investigates how different cores would affect the achieved level of fairness. Our experimental results show that the choice of density-ratio estimation core could significantly affect the outcome of fairness measurement method, and even, generate inconsistent results with respect to the relative fairness of various algorithms. These observations suggest major issues with density-ratio estimation based fairness measurement in regression and a need for further research to enhance their reliability.