FairJob: A Real-World Dataset for Fairness in Online Systems

TL;DR

FairJob introduces a real-world, privacy-compliant dataset for studying fairness in online job recommendation systems, enabling research on bias mitigation with practical implications.

Contribution

The paper presents a novel, anonymized dataset with proxy attributes for fairness research in advertising, along with methods to evaluate and improve fairness in biased datasets.

Findings

Bias mitigation techniques show potential to improve fairness.

Trade-offs exist between fairness and utility in recommendations.

The dataset provides a realistic benchmark for fairness research.

Abstract

We introduce a fairness-aware dataset for job recommendations in advertising, designed to foster research in algorithmic fairness within real-world scenarios. It was collected and prepared to comply with privacy standards and business confidentiality. An additional challenge is the lack of access to protected user attributes such as gender, for which we propose a solution to obtain a proxy estimate. Despite being anonymized and including a proxy for a sensitive attribute, our dataset preserves predictive power and maintains a realistic and challenging benchmark. This dataset addresses a significant gap in the availability of fairness-focused resources for high-impact domains like advertising -- the actual impact being having access or not to precious employment opportunities, where balancing fairness and utility is a common industrial challenge. We also explore various stages in the advertising process where unfairness can occur and introduce a method to compute a fair utility metric for the job recommendations in online systems case from a biased dataset. Experimental evaluations of bias mitigation techniques on the released dataset demonstrate potential improvements in fairness and the associated trade-offs with utility. The dataset is hosted at https://huggingface.co/datasets/criteo/FairJob. Source code for the experiments is hosted at https://github.com/criteo-research/FairJob-dataset/.