Model Mis-specification and Algorithmic Bias

TL;DR

This paper investigates how model mis-specification can lead to significant algorithmic bias, even with unbiased data, by analyzing group-level prediction errors and their dependence on feature distribution moments.

Contribution

It provides a theoretical analysis of bias caused by model mis-specification, revealing how group errors can differ despite similar overall error levels.

Findings

Group-level errors can be large despite negligible population error

Bias can be maximized with errors of opposite signs for different groups

Errors depend on the first and second moments of feature distributions

Abstract

Machine learning algorithms are increasingly used to inform critical decisions. There is a growing concern about bias, that algorithms may produce uneven outcomes for individuals in different demographic groups. In this work, we measure bias as the difference between mean prediction errors across groups. We show that even with unbiased input data, when a model is mis-specified: (1) population-level mean prediction error can still be negligible, but group-level mean prediction errors can be large; (2) such errors are not equal across groups; and (3) the difference between errors, i.e., bias, can take the worst-case realization. That is, when there are two groups of the same size, mean prediction errors for these two groups have the same magnitude but opposite signs. In closed form, we show such errors and bias are functions of the first and second moments of the joint distribution of features (for linear and probit regressions). We also conduct numerical experiments to show similar results in more general settings. Our work provides a first step for decoupling the impact of different causes of bias.