Scalable and adaptive prediction bands with kernel sum-of-squares

TL;DR

This paper introduces a scalable, adaptive conformal prediction method using kernel sum-of-squares and RKHS, improving coverage adaptivity and computational efficiency with a new hyperparameter tuning strategy.

Contribution

It extends conformal prediction with a dual formulation solvable by gradient methods and proposes a HSIC-based hyperparameter tuning for better adaptivity.

Findings

Efficient dual formulation enables handling hundreds to thousands of samples.

The HSIC-based tuning improves test-conditional coverage.

Experimental results show competitive performance with existing methods.

Abstract

Conformal Prediction (CP) is a popular framework for constructing prediction bands with valid coverage in finite samples, while being free of any distributional assumption. A well-known limitation of conformal prediction is the lack of adaptivity, although several works introduced practically efficient alternate procedures. In this work, we build upon recent ideas that rely on recasting the CP problem as a statistical learning problem, directly targeting coverage and adaptivity. This statistical learning problem is based on reproducible kernel Hilbert spaces (RKHS) and kernel sum-of-squares (SoS) methods. First, we extend previous results with a general representer theorem and exhibit the dual formulation of the learning problem. Crucially, such dual formulation can be solved efficiently by accelerated gradient methods with several hundreds or thousands of samples, unlike previous strategies based on off-the-shelf semidefinite programming algorithms. Second, we introduce a new hyperparameter tuning strategy tailored specifically to target adaptivity through bounds on test-conditional coverage. This strategy, based on the Hilbert-Schmidt Independence Criterion (HSIC), is introduced here to tune kernel lengthscales in our framework, but has broader applicability since it could be used in any CP algorithm where the score function is learned. Finally, extensive experiments are conducted to show how our method compares to related work. All figures can be reproduced with the accompanying code.