Putting Fairness Principles into Practice: Challenges, Metrics, and Improvements

TL;DR

This paper presents a case study on implementing fairness in a production machine learning system, introduces a new fairness metric called conditional equality, and proposes an improved training approach to enhance fairness in real-world applications.

Contribution

It offers practical insights into applying fairness metrics in production, introduces the conditional equality metric, and demonstrates an effective training method to improve fairness outcomes.

Findings

Conditional equality accounts for distributional differences in fairness assessment.

The proposed training approach improves fairness metrics in a real-world system.

The case study highlights challenges and solutions in deploying fairness in practice.

Abstract

As more researchers have become aware of and passionate about algorithmic fairness, there has been an explosion in papers laying out new metrics, suggesting algorithms to address issues, and calling attention to issues in existing applications of machine learning. This research has greatly expanded our understanding of the concerns and challenges in deploying machine learning, but there has been much less work in seeing how the rubber meets the road. In this paper we provide a case-study on the application of fairness in machine learning research to a production classification system, and offer new insights in how to measure and address algorithmic fairness issues. We discuss open questions in implementing equality of opportunity and describe our fairness metric, conditional equality, that takes into account distributional differences. Further, we provide a new approach to improve on the fairness metric during model training and demonstrate its efficacy in improving performance for a real-world product