Conformal Loss-Controlling Prediction

TL;DR

This paper introduces a novel conformal prediction framework that controls the loss for individual test instances, extending traditional conformal methods to a broader class of loss functions with proven guarantees.

Contribution

It extends conformal prediction to control arbitrary loss functions at the test level, providing finite-sample guarantees and broad applicability.

Findings

Framework effectively controls loss for individual test points.

Empirical tests show strong performance in classification and weather forecasting.

Theoretical guarantees hold under data exchangeability assumptions.

Abstract

Conformal prediction is a learning framework controlling prediction coverage of prediction sets, which can be built on any learning algorithm for point prediction. This work proposes a learning framework named conformal loss-controlling prediction, which extends conformal prediction to the situation where the value of a loss function needs to be controlled. Different from existing works about risk-controlling prediction sets and conformal risk control with the purpose of controlling the expected values of loss functions, the proposed approach in this paper focuses on the loss for any test object, which is an extension of conformal prediction from miscoverage loss to some general loss. The controlling guarantee is proved under the assumption of exchangeability of data in finite-sample cases and the framework is tested empirically for classification with a class-varying loss and statistical postprocessing of numerical weather forecasting applications, which are introduced as point-wise classification and point-wise regression problems. All theoretical analysis and experimental results confirm the effectiveness of our loss-controlling approach.