Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictions

TL;DR

This paper introduces a flexible framework for bounding the probability of high-loss predictions by focusing on quantiles of the loss distribution, providing rigorous guarantees for risk-sensitive applications.

Contribution

It proposes a novel quantile-based bounding framework that leverages order statistics, extending performance guarantees beyond expected loss to distributional risk measures.

Findings

Framework effectively controls loss quantiles on real datasets

Method applies to various quantile-based metrics

Theoretical analysis supports rigorous risk guarantees

Abstract

Rigorous guarantees about the performance of predictive algorithms are necessary in order to ensure their responsible use. Previous work has largely focused on bounding the expected loss of a predictor, but this is not sufficient in many risk-sensitive applications where the distribution of errors is important. In this work, we propose a flexible framework to produce a family of bounds on quantiles of the loss distribution incurred by a predictor. Our method takes advantage of the order statistics of the observed loss values rather than relying on the sample mean alone. We show that a quantile is an informative way of quantifying predictive performance, and that our framework applies to a variety of quantile-based metrics, each targeting important subsets of the data distribution. We analyze the theoretical properties of our proposed method and demonstrate its ability to rigorously control loss quantiles on several real-world datasets.