AutoCP: Automated Pipelines for Accurate Prediction Intervals

TL;DR

AutoCP is an AutoML framework that automates the construction of accurate, less conservative prediction intervals using conformal prediction, improving uncertainty quantification in machine learning applications.

Contribution

AutoCP introduces an automated approach to generate valid, optimized prediction intervals that meet target coverage rates, addressing over-conservativeness in conformal prediction.

Findings

AutoCP significantly outperforms benchmark algorithms in various datasets.

It constructs prediction intervals that meet specified coverage with reduced conservativeness.

The framework enhances the practical applicability of conformal prediction in real-world tasks.

Abstract

Successful application of machine learning models to real-world prediction problems, e.g. financial forecasting and personalized medicine, has proved to be challenging, because such settings require limiting and quantifying the uncertainty in the model predictions, i.e. providing valid and accurate prediction intervals. Conformal Prediction is a distribution-free approach to construct valid prediction intervals in finite samples. However, the prediction intervals constructed by Conformal Prediction are often (because of over-fitting, inappropriate measures of nonconformity, or other issues) overly conservative and hence inadequate for the application(s) at hand. This paper proposes an AutoML framework called Automatic Machine Learning for Conformal Prediction (AutoCP). Unlike the familiar AutoML frameworks that attempt to select the best prediction model, AutoCP constructs prediction intervals that achieve the user-specified target coverage rate while optimizing the interval length to be accurate and less conservative. We tested AutoCP on a variety of datasets and found that it significantly outperforms benchmark algorithms.