Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Making

TL;DR

This paper introduces a causal framework to analyze biases in proxy labels used in human-AI decision-making, highlighting the importance of understanding latent constructs and evaluating assumptions behind model training.

Contribution

It develops a causal framework to identify and disentangle sources of bias in proxy labels, guiding better evaluation and design of human-AI systems.

Findings

Most prior studies overlook target variable bias.

The framework clarifies assumptions in existing models.

Few studies evaluate bias-related factors explicitly.

Abstract

A growing literature on human-AI decision-making investigates strategies for combining human judgment with statistical models to improve decision-making. Research in this area often evaluates proposed improvements to models, interfaces, or workflows by demonstrating improved predictive performance on "ground truth" labels. However, this practice overlooks a key difference between human judgments and model predictions. Whereas humans reason about broader phenomena of interest in a decision -- including latent constructs that are not directly observable, such as disease status, the "toxicity" of online comments, or future "job performance" -- predictive models target proxy labels that are readily available in existing datasets. Predictive models' reliance on simplistic proxies makes them vulnerable to various sources of statistical bias. In this paper, we identify five sources of target variable bias that can impact the validity of proxy labels in human-AI decision-making tasks. We develop a causal framework to disentangle the relationship between each bias and clarify which are of concern in specific human-AI decision-making tasks. We demonstrate how our framework can be used to articulate implicit assumptions made in prior modeling work, and we recommend evaluation strategies for verifying whether these assumptions hold in practice. We then leverage our framework to re-examine the designs of prior human subjects experiments that investigate human-AI decision-making, finding that only a small fraction of studies examine factors related to target variable bias. We conclude by discussing opportunities to better address target variable bias in future research.