Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices

TL;DR

This paper investigates how companies develop, validate, and address bias in algorithmic hiring tools, revealing gaps between claims and practices, and discussing technical and legal challenges in bias mitigation.

Contribution

It provides a comprehensive analysis of vendor claims, development practices, and bias mitigation efforts in algorithmic employment assessments, highlighting gaps and challenges.

Findings

Many vendors disclose limited validation details

Bias mitigation techniques vary widely across vendors

Legal and technical challenges complicate bias reduction

Abstract

There has been rapidly growing interest in the use of algorithms in hiring, especially as a means to address or mitigate bias. Yet, to date, little is known about how these methods are used in practice. How are algorithmic assessments built, validated, and examined for bias? In this work, we document and analyze the claims and practices of companies offering algorithms for employment assessment. In particular, we identify vendors of algorithmic pre-employment assessments (i.e., algorithms to screen candidates), document what they have disclosed about their development and validation procedures, and evaluate their practices, focusing particularly on efforts to detect and mitigate bias. Our analysis considers both technical and legal perspectives. Technically, we consider the various choices vendors make regarding data collection and prediction targets, and explore the risks and trade-offs that these choices pose. We also discuss how algorithmic de-biasing techniques interface with, and create challenges for, antidiscrimination law.