LiFT: A Scalable Framework for Measuring Fairness in ML Applications

TL;DR

LiFT is a scalable framework designed to measure and address fairness in large-scale machine learning systems, helping mitigate societal biases in web applications.

Contribution

The paper introduces LiFT, a novel scalable toolkit for measuring fairness metrics in large ML systems, with practical deployment insights from LinkedIn.

Findings

Successfully deployed at LinkedIn for real-world fairness measurement

Identified challenges and lessons in integrating fairness tools at scale

Provides open problems for future research in fairness measurement

Abstract

Many internet applications are powered by machine learned models, which are usually trained on labeled datasets obtained through either implicit / explicit user feedback signals or human judgments. Since societal biases may be present in the generation of such datasets, it is possible for the trained models to be biased, thereby resulting in potential discrimination and harms for disadvantaged groups. Motivated by the need for understanding and addressing algorithmic bias in web-scale ML systems and the limitations of existing fairness toolkits, we present the LinkedIn Fairness Toolkit (LiFT), a framework for scalable computation of fairness metrics as part of large ML systems. We highlight the key requirements in deployed settings, and present the design of our fairness measurement system. We discuss the challenges encountered in incorporating fairness tools in practice and the lessons learned during deployment at LinkedIn. Finally, we provide open problems based on practical experience.