Conformal Calibration of Statistical Confidence Sets

TL;DR

This paper introduces TRUST and TRUST++ methods that calibrate confidence sets using simulated data, ensuring valid coverage in complex models and small samples, bridging conformal prediction with statistical inference.

Contribution

The paper presents novel calibration methods, TRUST and TRUST++, that achieve distribution-free conditional coverage using only simulated data, applicable to complex and intractable models.

Findings

Methods ensure finite-sample local coverage.

They achieve asymptotic conditional coverage with increasing simulations.

Outperform existing approaches in small-sample regimes.

Abstract

Constructing valid confidence sets is a crucial task in statistical inference, yet traditional methods often face challenges when dealing with complex models or limited observed sample sizes. These challenges are frequently encountered in modern applications, such as Likelihood-Free Inference (LFI). In these settings, confidence sets may fail to maintain a confidence level close to the nominal value. In this paper, we introduce two novel methods, TRUST and TRUST++, for calibrating confidence sets to achieve distribution-free conditional coverage. These methods rely entirely on simulated data from the statistical model to perform calibration. Leveraging insights from conformal prediction techniques adapted to the statistical inference context, our methods ensure both finite-sample local coverage and asymptotic conditional coverage as the number of simulations increases, even if n is small. They effectively handle nuisance parameters and provide computationally efficient uncertainty quantification for the estimated confidence sets. This allows users to assess whether additional simulations are necessary for robust inference. Through theoretical analysis and experiments on models with both tractable and intractable likelihoods, we demonstrate that our methods outperform existing approaches, particularly in small-sample regimes. This work bridges the gap between conformal prediction and statistical inference, offering practical tools for constructing valid confidence sets in complex models.