Conformal Risk Control for Non-Monotonic Losses

TL;DR

This paper extends conformal risk control to handle non-monotonic losses with multidimensional parameters, providing stability-dependent guarantees and applications in medical imaging, fairness, and classification tasks.

Contribution

It introduces risk control guarantees for non-monotonic, multidimensional loss functions, broadening the applicability of conformal prediction methods.

Findings

Guarantees depend on algorithm stability

Applications include image classification and fairness in recidivism prediction

Demonstrates effectiveness in diverse real-world tasks

Abstract

Conformal risk control is an extension of conformal prediction for controlling risk functions beyond miscoverage. The original algorithm controls the expected value of a loss that is monotonic in a one-dimensional parameter. Here, we present risk control guarantees for generic algorithms applied to possibly non-monotonic losses with multidimensional parameters. The guarantees depend on the stability of the algorithm -- unstable algorithms have looser guarantees. We give applications of this technique to selective image classification, FDR and IOU control of tumor segmentations, and multigroup debiasing of recidivism predictions across overlapping race and sex groups using empirical risk minimization.