Nested Conformal Prediction Sets for Classification with Applications to Probation Data

TL;DR

This paper introduces nested conformal prediction sets to improve uncertainty quantification in risk assessments for criminal justice, demonstrating their advantages over standard methods using probation data.

Contribution

The paper presents a novel application of nested conformal prediction sets and a localized conformal method for more accurate uncertainty estimation in criminal risk forecasts.

Findings

Nested conformal prediction sets can significantly differ from standard uncertainty measures.

Localized conformal method improves alignment with confusion tables.

The proposed methods show promise for fairer and more accurate risk assessments.

Abstract

Risk assessments to help inform criminal justice decisions have been used in the United States since the 1920s. Over the past several years, statistical learning risk algorithms have been introduced amid much controversy about fairness, transparency and accuracy. In this paper, we focus on accuracy for a large department of probation and parole that is considering a major revision of its current, statistical learning risk methods. Because the content of each offender's supervision is substantially shaped by a forecast of subsequent conduct, forecasts have real consequences. Here we consider the probability that risk forecasts are correct. We augment standard statistical learning estimates of forecasting uncertainty (i.e., confusion tables) with uncertainty estimates from nested conformal prediction sets. In a demonstration of concept using data from the department of probation and parole, we show that the standard uncertainty measures and uncertainty measures from nested conformal prediction sets can differ dramatically in concept and output. We also provide a modification of nested conformal called the localized conformal method to match confusion tables more closely when possible. A strong case can be made favoring the nested and localized conformal approach. As best we can tell, our formulation of such comparisons and consequent recommendations is novel.