Systematic Bias in Sample Inference and its Effect on Machine Learning

TL;DR

This paper identifies that systematic bias in machine learning, especially underprediction for minorities, arises from small-sample statistical inference, and demonstrates this bias through analysis of decision tree models on real datasets.

Contribution

It reveals that small-sample inference causes systematic bias in ML predictions, explaining underprediction patterns for minority groups, supported by empirical analysis.

Findings

Bias correlates strongly with small-sample inference in models

Underprediction is more severe for minority groups due to sample size effects

Small-sample inference bias explains observed underprediction patterns

Abstract

A commonly observed pattern in machine learning models is an underprediction of the target feature, with the model's predicted target rate for members of a given category typically being lower than the actual target rate for members of that category in the training set. This underprediction is usually larger for members of minority groups; while income level is underpredicted for both men and women in the 'adult' dataset, for example, the degree of underprediction is significantly higher for women (a minority in that dataset). We propose that this pattern of underprediction for minorities arises as a predictable consequence of statistical inference on small samples. When presented with a new individual for classification, an ML model performs inference not on the entire training set, but on a subset that is in some way similar to the new individual, with sizes of these subsets typically following a power law distribution so that most are small (and with these subsets being necessarily smaller for the minority group). We show that such inference on small samples is subject to systematic and directional statistical bias, and that this bias produces the observed patterns of underprediction seen in ML models. Analysing a standard sklearn decision tree model's predictions on a set of over 70 subsets of the 'adult' and COMPAS datasets, we found that a bias prediction measure based on small-sample inference had a significant positive correlations (0.56 and 0.85) with the observed underprediction rate for these subsets.