Confronting Label Indeterminacy in Automated Bail Decisions

TL;DR

This paper examines how to handle uncertain labels in machine learning models for bail decisions, highlighting the impact of different approaches on model behavior and legal legitimacy.

Contribution

It introduces a novel label imputation method and evaluates five approaches to address label indeterminacy in bail decision data.

Findings

All methods influence model predictions significantly.

Model choice impacts results more than label handling approach.

Explainable AI reveals effects on internal decision processes.

Abstract

Bail decisions present a fundamental challenge for data-driven decision support systems. When bail is denied, the counterfactual outcome of whether the defendant would have appeared in court remains unobserved. As a result, historical bail data embed structural label indeterminacy: future decisions are influenced by past decisions whose outcomes are only partially knowable. Building automated systems on such data risks introducing bias and reinforcing feedback loops. This raises a core question for machine-learning systems intended to assist judicial actors: how should cases in which bail was denied be treated during model development? In a case study of bail decisions from the Unified Judicial System of Pennsylvania, we evaluate five contemporary approaches to handling label indeterminacy across three machine learning models, including a novel label imputation method motivated by the dynamics of bail decisions. Each method relies on unverifiable assumptions, yet all influence the models' predictive behaviour, sometimes even more so than the choice of model itself. Explainable AI analysis further reveals that these effects extend to the models' internal decision-making processes as well. Finally, we consider the notion of label indeterminacy from a legal perspective and assess the legitimacy of these approaches in the context of bail decision-making.