50 Years of Test (Un)fairness: Lessons for Machine Learning

TL;DR

This paper reviews 50 years of fairness concepts in testing, showing their evolution and relevance to modern machine learning fairness, highlighting overlooked insights and guiding future research directions.

Contribution

It provides a comprehensive historical analysis of fairness definitions across disciplines, linking past ideas to current machine learning fairness challenges.

Findings

Historical fairness definitions often align with modern concepts

Past insights can inform current fairness measurement methods

Understanding social context enhances fairness assessments

Abstract

Quantitative definitions of what is unfair and what is fair have been introduced in multiple disciplines for well over 50 years, including in education, hiring, and machine learning. We trace how the notion of fairness has been defined within the testing communities of education and hiring over the past half century, exploring the cultural and social context in which different fairness definitions have emerged. In some cases, earlier definitions of fairness are similar or identical to definitions of fairness in current machine learning research, and foreshadow current formal work. In other cases, insights into what fairness means and how to measure it have largely gone overlooked. We compare past and current notions of fairness along several dimensions, including the fairness criteria, the focus of the criteria (e.g., a test, a model, or its use), the relationship of fairness to individuals, groups, and subgroups, and the mathematical method for measuring fairness (e.g., classification, regression). This work points the way towards future research and measurement of (un)fairness that builds from our modern understanding of fairness while incorporating insights from the past.