CAOS: Conformal Aggregation of One-Shot Predictors

TL;DR

CAOS is a new conformal prediction framework that adaptively combines multiple one-shot predictors with a leave-one-out calibration scheme, providing valid uncertainty quantification and smaller prediction sets in scarce data scenarios.

Contribution

It introduces CAOS, a novel conformal aggregation method for one-shot predictors that achieves valid coverage despite violating classical assumptions.

Findings

CAOS produces smaller prediction sets than baseline methods.

CAOS maintains reliable coverage in experiments.

Effective in facial landmarking and text classification tasks.

Abstract

One-shot prediction enables rapid adaptation of pretrained foundation models to new tasks using only one labeled example, but lacks principled uncertainty quantification. While conformal prediction provides finite-sample coverage guarantees, standard split conformal methods are inefficient in the one-shot setting due to data splitting and reliance on a single predictor. We propose Conformal Aggregation of One-Shot Predictors (CAOS), a conformal framework that adaptively aggregates multiple one-shot predictors and uses a leave-one-out calibration scheme to fully exploit scarce labeled data. Despite violating classical exchangeability assumptions, we prove that CAOS achieves valid marginal coverage using a monotonicity-based argument. Experiments on one-shot facial landmarking and RAFT text classification tasks show that CAOS produces substantially smaller prediction sets than split conformal baselines while maintaining reliable coverage.