Loss-Controlling Calibration for Predictive Models

TL;DR

This paper introduces a flexible calibration framework for predictive models that controls loss values directly, extending conformal prediction to handle general loss functions and non-set predictors with theoretical guarantees.

Contribution

It develops a loss-controlling calibration method that applies to any measurable loss function and non-nested predictors, with finite-sample guarantees and broad applicability.

Findings

Effective in selective regression tasks

Demonstrated success in high-impact weather forecasting

Provides finite-sample control guarantees

Abstract

We propose a learning framework for calibrating predictive models to make loss-controlling prediction for exchangeable data, which extends our recently proposed conformal loss-controlling prediction for more general cases. By comparison, the predictors built by the proposed loss-controlling approach are not limited to set predictors, and the loss function can be any measurable function without the monotone assumption. To control the loss values in an efficient way, we introduce transformations preserving exchangeability to prove finite-sample controlling guarantee when the test label is obtained, and then develop an approximation approach to construct predictors. The transformations can be built on any predefined function, which include using optimization algorithms for parameter searching. This approach is a natural extension of conformal loss-controlling prediction, since it can be reduced to the latter when the set predictors have the nesting property and the loss functions are monotone. Our proposed method is applied to selective regression and high-impact weather forecasting problems, which demonstrates its effectiveness for general loss-controlling prediction.