Technical Note: Bias and the Quantification of Stability

TL;DR

This paper emphasizes the importance of stability in machine learning bias evaluation, proposing a method to quantify stability through concept agreement and exploring its relationship with accuracy and bias.

Contribution

It introduces a novel method for measuring algorithm stability based on concept agreement, expanding bias assessment beyond predictive accuracy.

Findings

Proposes a stability quantification method based on concept agreement

Highlights the relationship between stability, accuracy, and bias

Encourages broader evaluation criteria for bias in machine learning

Abstract

Research on bias in machine learning algorithms has generally been concerned with the impact of bias on predictive accuracy. We believe that there are other factors that should also play a role in the evaluation of bias. One such factor is the stability of the algorithm; in other words, the repeatability of the results. If we obtain two sets of data from the same phenomenon, with the same underlying probability distribution, then we would like our learning algorithm to induce approximately the same concepts from both sets of data. This paper introduces a method for quantifying stability, based on a measure of the agreement between concepts. We also discuss the relationships among stability, predictive accuracy, and bias.