End-to-End Conformal Calibration for Optimization Under Uncertainty

TL;DR

This paper introduces an end-to-end conformal calibration framework that learns uncertainty sets tailored for decision-making under uncertainty, improving robustness and calibration in high-dimensional settings.

Contribution

It develops a novel end-to-end method that learns uncertainty sets informed by decision loss, using conformal prediction for guarantees, and employs input-convex neural networks for flexible uncertainty modeling.

Findings

Improves robustness and calibration guarantees over baseline methods.

Enhances decision-making performance in energy storage and portfolio optimization.

Provides a scalable approach for high-dimensional uncertainty estimation.

Abstract

Machine learning can significantly improve performance for decision-making under uncertainty across a wide range of domains. However, ensuring robustness guarantees requires well-calibrated uncertainty estimates, which can be difficult to achieve with neural networks. Moreover, in high-dimensional settings, there may be many valid uncertainty estimates, each with its own performance profile - i.e., not all uncertainty is equally valuable for downstream decision-making. To address this problem, this paper develops an end-to-end framework to learn uncertainty sets for conditional robust optimization in a way that is informed by the downstream decision-making loss, with robustness and calibration guarantees provided by conformal prediction. In addition, we propose to represent general convex uncertainty sets with partially input-convex neural networks, which are learned as part of our framework. Our approach consistently improves upon two-stage estimate-then-optimize baselines on concrete applications in energy storage arbitrage and portfolio optimization.