Feedback Effects in Repeat-Use Criminal Risk Assessments

TL;DR

This paper investigates how feedback effects in sequential criminal risk assessments can amplify biases over time, revealing limitations of current validation methods that rely on one-shot testing.

Contribution

It introduces a Polya Urn model to simulate feedback effects, demonstrating how small biases can compound in risk assessment processes over multiple iterations.

Findings

Biases can amplify over sequential decisions

One-shot validation tests may overlook long-term bias propagation

Risk assessment tools need new development and auditing approaches

Abstract

In the criminal legal context, risk assessment algorithms are touted as data-driven, well-tested tools. Studies known as validation tests are typically cited by practitioners to show that a particular risk assessment algorithm has predictive accuracy, establishes legitimate differences between risk groups, and maintains some measure of group fairness in treatment. To establish these important goals, most tests use a one-shot, single-point measurement. Using a Polya Urn model, we explore the implication of feedback effects in sequential scoring-decision processes. We show through simulation that risk can propagate over sequential decisions in ways that are not captured by one-shot tests. For example, even a very small or undetectable level of bias in risk allocation can amplify over sequential risk-based decisions, leading to observable group differences after a number of decision iterations. Risk assessment tools operate in a highly complex and path-dependent process, fraught with historical inequity. We conclude from this study that these tools do not properly account for compounding effects, and require new approaches to development and auditing.