Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction

TL;DR

This paper introduces meta-XB, a meta-learning approach that improves the calibration and reduces the size of set predictors in conformal prediction, especially with limited data, while maintaining formal guarantees.

Contribution

The paper proposes meta-XB, a novel meta-learning scheme using cross-validation-based conformal prediction that preserves calibration guarantees and reduces set size in low-data scenarios.

Findings

Meta-XB effectively reduces set size compared to traditional conformal prediction.

The method maintains formal per-task calibration guarantees.

Adaptive non-conformal scores further improve calibration accuracy.

Abstract

Conventional frequentist learning is known to yield poorly calibrated models that fail to reliably quantify the uncertainty of their decisions. Bayesian learning can improve calibration, but formal guarantees apply only under restrictive assumptions about correct model specification. Conformal prediction (CP) offers a general framework for the design of set predictors with calibration guarantees that hold regardless of the underlying data generation mechanism. However, when training data are limited, CP tends to produce large, and hence uninformative, predicted sets. This paper introduces a novel meta-learning solution that aims at reducing the set prediction size. Unlike prior work, the proposed meta-learning scheme, referred to as meta-XB, (i) builds on cross-validation-based CP, rather than the less efficient validation-based CP; and (ii) preserves formal per-task calibration guarantees, rather than less stringent task-marginal guarantees. Finally, meta-XB is extended to adaptive non-conformal scores, which are shown empirically to further enhance marginal per-input calibration.