Safe Policy Learning through Extrapolation: Application to Pre-trial Risk Assessment

TL;DR

This paper introduces a robust optimization method to improve deterministic pre-trial risk assessment policies in the US criminal justice system, ensuring safety and limited downside risk while enhancing certain risk classifications.

Contribution

It develops a maximin robust optimization approach for policy learning with deterministic policies, addressing the challenge of policy improvement without stochasticity.

Findings

Certain risk assessment components can be safely improved.

The method provides a safety guarantee under structural assumptions.

Some components remain inconclusive for improvement.

Abstract

Algorithmic recommendations and decisions have become ubiquitous in today's society. Many of these data-driven policies, especially in the realm of public policy, are based on known, deterministic rules to ensure their transparency and interpretability. We examine a particular case of algorithmic pre-trial risk assessments in the US criminal justice system, which provide deterministic classification scores and recommendations to help judges make release decisions. Our goal is to analyze data from a unique field experiment on an algorithmic pre-trial risk assessment to investigate whether the scores and recommendations can be improved. Unfortunately, prior methods for policy learning are not applicable because they require existing policies to be stochastic. We develop a maximin robust optimization approach that partially identifies the expected utility of a policy, and then finds a policy that maximizes the worst-case expected utility. The resulting policy has a statistical safety property, limiting the probability of producing a worse policy than the existing one, under structural assumptions about the outcomes. Our analysis of data from the field experiment shows that we can safely improve certain components of the risk assessment instrument by classifying arrestees as lower risk under a wide range of utility specifications, though the analysis is not informative about several components of the instrument.