Calibrated inference: statistical inference that accounts for both sampling uncertainty and distributional uncertainty

TL;DR

This paper introduces a new statistical inference method called calibrated inference that constructs confidence intervals accounting for both sampling and distributional uncertainties, enhancing the trustworthiness of scientific conclusions.

Contribution

It formalizes a distributional uncertainty model and develops a stability-based approach to create confidence intervals that incorporate multiple sources of uncertainty.

Findings

Confidence intervals that account for distributional shifts

The distributional perturbation model under symmetry assumptions

Stability analysis enables robust inference

Abstract

How can we draw trustworthy scientific conclusions? One criterion is that a study can be replicated by independent teams. While replication is critically important, it is arguably insufficient. If a study is biased for some reason and other studies recapitulate the approach then findings might be consistently incorrect. It has been argued that trustworthy scientific conclusions require disparate sources of evidence. However, different methods might have shared biases, making it difficult to judge the trustworthiness of a result. We formalize this issue by introducing a "distributional uncertainty model", wherein dense distributional shifts emerge as the superposition of numerous small random changes. The distributional perturbation model arises under a symmetry assumption on distributional shifts and is strictly weaker than assuming that the data is i.i.d. from the target distribution. We show that a stability analysis on a single data set allows us to construct confidence intervals that account for both sampling uncertainty and distributional uncertainty.