Distribution-free inference for regression: discrete, continuous, and in between

TL;DR

This paper explores distribution-free inference for regression across different types of feature distributions, identifying regimes where confidence intervals can have vanishing width based on the support size relative to sample size.

Contribution

It characterizes the regimes between finite and continuous feature distributions where distribution-free confidence intervals with vanishing width are possible.

Findings

Vanishing-width confidence intervals are achievable when the effective support size is less than the square of the sample size.

In continuous feature settings, confidence intervals must have non-vanishing width.

The paper delineates distinct regimes based on the support size of the feature distribution.

Abstract

In data analysis problems where we are not able to rely on distributional assumptions, what types of inference guarantees can still be obtained? Many popular methods, such as holdout methods, cross-validation methods, and conformal prediction, are able to provide distribution-free guarantees for predictive inference, but the problem of providing inference for the underlying regression function (for example, inference on the conditional mean $\mathbb{E}[Y|X]$) is more challenging. In the setting where the features $X$ are continuously distributed, recent work has established that any confidence interval for $\mathbb{E}[Y|X]$ must have non-vanishing width, even as sample size tends to infinity. At the other extreme, if $X$ takes only a small number of possible values, then inference on $\mathbb{E}[Y|X]$ is trivial to achieve. In this work, we study the problem in settings in between these two extremes. We find that there are several distinct regimes in between the finite setting and the continuous setting, where vanishing-width confidence intervals are achievable if and only if the effective support size of the distribution of $X$ is smaller than the square of the sample size.