Data-Adaptive Tradeoffs among Multiple Risks in Distribution-Free Prediction

TL;DR

This paper introduces data-adaptive methods for managing multiple risks in distribution-free prediction, ensuring valid risk control without distributional assumptions, demonstrated through synthetic and real-world vision data.

Contribution

It develops novel techniques for adaptive risk tradeoff management in distribution-free settings, addressing limitations of existing uncertainty quantification methods.

Findings

Valid risk control achieved with adaptive threshold selection

Method supports monotone and nearly-monotone risks

Effective on synthetic and large-scale vision datasets

Abstract

Decision-making pipelines are generally characterized by tradeoffs among various risk functions. It is often desirable to manage such tradeoffs in a data-adaptive manner. As we demonstrate, if this is done naively, state-of-the art uncertainty quantification methods can lead to significant violations of putative risk guarantees. To address this issue, we develop methods that permit valid control of risk when threshold and tradeoff parameters are chosen adaptively. Our methodology supports monotone and nearly-monotone risks, but otherwise makes no distributional assumptions. To illustrate the benefits of our approach, we carry out numerical experiments on synthetic data and the large-scale vision dataset MS-COCO.