Long-term Fairness with Selective Labels

TL;DR

This paper addresses long-term fairness in decision-making scenarios with selective labels, proposing a new framework and reinforcement learning algorithm to ensure fairness despite label limitations.

Contribution

It introduces a novel framework combining observed data and label prediction to estimate true fairness, along with a reinforcement learning algorithm for fair decision-making.

Findings

The proposed algorithm achieves fairness comparable to oracle-based methods in semisynthetic tests.

Naive solutions do not guarantee fairness in selective label settings.

The framework provides sufficient conditions to ensure true fairness from observable data.

Abstract

Long-term fairness algorithms aim to satisfy fairness beyond static and short-term notions by accounting for the dynamics between decision-making policies and population behavior. Most previous approaches evaluate performance and fairness measures from observable features and a label, which is assumed to be fully observed. However, in scenarios such as hiring or lending, the labels (e.g., ability to repay the loan) are selective labels as they are only revealed based on positive decisions (e.g., when a loan is granted). In this paper, we study long-term fairness in the selective labels setting and analytically show that naive solutions do not guarantee fairness. To address this gap, we then introduce a novel framework that leverages both the observed data and a label predictor model to estimate the true fairness measure value by decomposing it into the observed fairness and bias from label predictions. This allows us to derive sufficient conditions to satisfy true fairness from observable quantities by using the confidence in the predictor model. Finally, we rely on our theoretical results to propose a novel reinforcement learning algorithm for effective long-term fair decision-making with selective labels. In semisynthetic environments, the proposed algorithm reached comparable fairness and performance to an agent with oracle access to the true labels.