Predictive inference with the jackknife+
Rina Foygel Barber, Emmanuel J. Candes, Aaditya Ramdas, Ryan J., Tibshirani
TL;DR
The paper introduces the jackknife+ method for constructing predictive confidence intervals with guaranteed coverage, improving upon the original jackknife by accounting for variability in predictions, and extends it to K-fold cross-validation.
Contribution
The paper proposes the jackknife+ method, providing rigorous coverage guarantees for predictive intervals and extending it to K-fold cross-validation, with theoretical and empirical validation.
Findings
Jackknife+ achieves nearly exact coverage.
Intervals are similar in length to the original jackknife under stability.
Coverage guarantees hold regardless of data distribution.
Abstract
This paper introduces the jackknife+, which is a novel method for constructing predictive confidence intervals. Whereas the jackknife outputs an interval centered at the predicted response of a test point, with the width of the interval determined by the quantiles of leave-one-out residuals, the jackknife+ also uses the leave-one-out predictions at the test point to account for the variability in the fitted regression function. Assuming exchangeable training samples, we prove that this crucial modification permits rigorous coverage guarantees regardless of the distribution of the data points, for any algorithm that treats the training points symmetrically. Such guarantees are not possible for the original jackknife and we demonstrate examples where the coverage rate may actually vanish. Our theoretical and empirical analysis reveals that the jackknife and the jackknife+ intervals…
| Method | Assumption-free theory | Typical empirical coverage |
|---|---|---|
| Naive (4) | No guarantee | |
| Split conf. (holdout) (3) | coverage | |
| Jackknife (7) | No guarantee | , or if unstable |
| Jackknife+ (9) | coverage | |
| Jackknife-minmax (10) | coverage | |
| Full conformal (15) | coverage | , or if overfits |
| K-fold CV+ (11) | coverage | |
| K-fold cross-conf. (12) | coverage |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Predictive inference with the jackknife+
Rina Foygel Barber , Emmanuel J. Candès ,
Aaditya Ramdas , Ryan J. Tibshirani††footnotemark: Department of Statistics, University of ChicagoDepartments of Statistics and Mathematics, Stanford UniversityDepartment of Statistics and Data Science, Carnegie Mellon UniversityMachine Learning Department, Carnegie Mellon University
Abstract
This paper introduces the jackknife+, which is a novel method for constructing predictive confidence intervals. Whereas the jackknife outputs an interval centered at the predicted response of a test point, with the width of the interval determined by the quantiles of leave-one-out residuals, the jackknife+ also uses the leave-one-out predictions at the test point to account for the variability in the fitted regression function. Assuming exchangeable training samples, we prove that this crucial modification permits rigorous coverage guarantees regardless of the distribution of the data points, for any algorithm that treats the training points symmetrically. Such guarantees are not possible for the original jackknife and we demonstrate examples where the coverage rate may actually vanish. Our theoretical and empirical analysis reveals that the jackknife and the jackknife+ intervals achieve nearly exact coverage and have similar lengths whenever the fitting algorithm obeys some form of stability. Further, we extend the jackknife+ to -fold cross validation and similarly establish rigorous coverage properties. Our methods are related to cross-conformal prediction proposed by Vovk (2015) and we discuss connections.
1 Introduction
Suppose that we have i.i.d. training data , , and a new test point drawn independently from the same distribution. We would like to fit a regression model to the training data, i.e., a function where predicts given a new feature vector , and then provide a prediction interval for the test point—an interval around that is likely to contain the true test response value . Specifically, given some target coverage level , we would like to construct a prediction interval as a function of the training data points, such that
[TABLE]
where the probability is taken with respect to a new test point as well as with respect to the training data.
A naive solution might be to use the residuals on the training data, , to estimate the typical prediction error on the new test point—for instance, we might consider the prediction interval
[TABLE]
However, in practice, this interval would typically undercover (meaning that the probability that lies in this interval would be lower than the target level ), since due to overfitting, the residuals on the training data points are typically smaller than the residual on the previously unseen test point, i.e., .
In order to avoid the overfitting problem, the jackknife prediction method computes a margin of error with a leave-one-out construction:
- •
For each , fit the regression function to the training data with the th point removed, and compute the corresponding leave-one-out residual, .
- •
Fit the regression function to the full training data, and output the prediction interval
[TABLE]
Intuitively, this method should have the right coverage properties on average since it avoids overfitting—the leave-one-out residuals reflect the typical magnitude of the error in predicting a new data point after fitting to a sample size (or, almost equivalently, ), unlike the naive method where the residuals on the training data are likely to be too small due to overfitting.
However, the jackknife procedure does not have any universal theoretical guarantees. Although many results are known under asymptotic settings or under assumptions of stability of the regression algorithm (we will give an overview below), it is nonetheless the case that, for settings where is unstable, the jackknife method may lose predictive coverage—for example, we will see in our simulations in Section 7 that the jackknife can have extremely poor coverage using least squares regression when the sample size is close to the dimension .
In this paper, we introduce a new method, the jackknife+, that provides non-asymptotic coverage guarantees under no assumptions beyond the training and test data being exchangeable. We will see that the jackknife+ offers, in the worst case, a coverage rate (where is the target), while the original jackknife may even have zero coverage in degenerate examples. On the other hand, empirically we often observe that the two methods yield nearly identical intervals and both achieve coverage. Theoretically, we will see that under a suitable notion of stability, the jackknife+ and jackknife both provably yield close to coverage.
1.1 Background
The idea of resampling or subsampling from the available data, in order to assess the accuracy of our parameter estimates or predictions, has a rich history in the statistics literature. Early works developing the jackknife and bootstrap methods include Quenouille (1949, 1956); Tukey (1958); Miller (1974); Efron (1979); Stine (1985). Several papers from this period include leave-one-out methods for assessing or calibrating predictive accuracy, similar to the predictive interval constructed in (2) above, e.g., Stone (1974); Geisser (1975); Butler and Rothman (1980), generally using the term “cross-validation” to refer to this approach. (In this work, we will instead use the term “jackknife” to refer to the leave-one-out style of prediction methods, as is common in the modern literature.) Efron and Gong (1983) provides an overview of the early literature on these types of methods.
While this rich literature demonstrated extensive evidence of the reliable performance of the jackknife in practice, relatively little has been known about the theoretical properties of this type of method until recently. Steinberger and Leeb (2016, 2018) have developed results proving valid predictive coverage of the jackknife under assumptions of algorithmic stability, meaning that the fitted model and its leave-one-out version are required to give similar predictions at the test point. This work builds on earlier results by Bousquet and Elisseeff (2002), which study generalization bounds for risk minimization through the framework of stability conditions; an earlier work in this line is that of Devroye and Wagner (1979), which give analogous results for classification risk.
In contrast to cross-validation methods, which perform well but are difficult to analyze theoretically, we can instead consider a simple validation or holdout method. We first partition the training data as , then fit on the subset of the training data and construct a predictive interval
[TABLE]
Papadopoulos (2008); Vovk (2012); Lei et al. (2018) study this type of method, under the name split conformal prediction or inductive conformal prediction, through the framework of exchangeability, and prove that predictive coverage holds with no assumptions on the algorithm or on the distribution of the data (with a small correction to the definition of the quantile). This method is also computationally very cheap, as we only need to fit a single regression function —in contrast, jackknife and cross-validation methods require running the regression many times. However, these benefits come at a statistical cost. If the training size is much smaller than , then the fitted model may be a poor fit, leading to wide prediction intervals; if instead we decide to take then instead is very small, leading to high variability.
Finally, Vovk (2015); Vovk et al. (2018) proposed the cross-conformal prediction method, which is closely related to the jackknife+. We describe the cross-conformal method, and the previously known theoretical guarantees, in detail later on. Their work is based on the conformal prediction method (see Vovk et al. (2005); Lei et al. (2018) for background), which provably achieves distribution-free predictive coverage at the target level but at an extremely high computational cost.
1.2 Notation
Before proceeding, we first define some notation. First, for any values indexed by , define111 In defining the quantiles of the residuals, we use rather than to correct for the finite sample size—we will see later on why this correction is natural. For the jackknife, it is perhaps more common to see in place of , i.e., the residual quantile is defined slightly differently, but for large the difference is negligible. Formally, if and so , then we set .
[TABLE]
the quantile of the empirical distribution of these values. Similarly, we will let denote the quantile of the distribution,
[TABLE]
With this notation, the “naive” prediction interval in (1) can be defined as
[TABLE]
Second, we will write to denote the algorithm mapping a training data set of any size, to the fitted regression function. Formally, is a map from \cup_{m\geq 1}\big{(}\mathbb{R}^{d}\times\mathbb{R}\big{)}^{m} (i.e., the collection of all training sets of any size ), to the space of functions . For example, when is the regression function fitted on the training data , we can write
[TABLE]
Similarly, to compute the leave-one-out residuals for the jackknife, we let
[TABLE]
and then the jackknife prediction interval (2) can be written as
[TABLE]
where denotes the th leave-one-out residual.
From this point on, we will assume without comment that satisfies a symmetry condition, namely, must be invariant to reordering the data, i.e.,
[TABLE]
for any sample size , any points , and any permutation of the indices .
2 The jackknife+
Our jackknife+ method is a modification of the jackknife (7). Defining as in (6), the jackknife+ prediction interval is given by:
[TABLE]
To compare this to the usual jackknife, we observe that can equivalently be defined as
[TABLE]
The constructions of the usual jackknife and the new jackknife+ are compared in Figure 1. While both versions of jackknife use the leave-one-out residuals, the difference is that for jackknife, we center our interval on the predicted value fitted on the full training data, while for jackknife+ we use the leave-one-out predictions for the test point.
Figure 1 illustrates that, if the leave-one-out fitted functions are all quite similar to , which was fitted on the full training data, then the two methods should return nearly identical prediction intervals. On the other hand, in settings where the regression algorithm is extremely sensitive to the training data, such that removing one data point can substantially change the predicted value at , the output may be quite different. In Section 5, we will examine the role of this type of instability in more closely.
To give one further interpretation of the difference between the two methods, while the jackknife interval is defined as a symmetric interval around the prediction for the test point (7), the jackknife+ interval can be interpreted as an interval around the median prediction,
[TABLE]
which is guaranteed to lie inside for (in general, however, the jackknife+ interval will not be symmetric around this median prediction).
As detailed in Section 7, the jackknife and jackknife+ often perform nearly identically in practice (and generally achieve an empirical coverage level very close to the target ), but in some more challenging examples where the regression algorithm is less stable, the original jackknife may lose coverage while jackknife+ still achieves the target coverage level.
Finally, we remark that in settings where the distribution of is highly skewed, it may be more natural to consider an asymmetric version of this method; we consider this extension in Appendix A.
2.1 Assumption-free guarantees
Remarkably, although the jackknife+ method appears to only be a slight modification of jackknife, our main result proves that the jackknife+ is guaranteed to achieve predictive coverage at the level , without making any assumptions on the distribution of the data or the nature of the regression method .
For this theorem, and all results that follow, all probabilities are stated with respect to the distribution of the training data points , and the test data point drawn i.i.d. from an arbitrary distribution , and we assume implicitly that the regression method is invariant to the ordering of the data (8). We will treat the sample size and the target coverage level as fixed throughout.
Theorem 1**.**
The jackknife+ prediction interval satisfies
[TABLE]
This result is proved in Section 6 using the exchangeability of the data points —we remark that this theorem actually holds more generally under the assumption that the data points are exchangeable, with the i.i.d. assumption as a special case.
In practice, we generally expect to achieve the target level with either version of the jackknife. A natural question is whether the factor of 2 appearing in the coverage guarantee for jackknife+ is real, or is merely an artifact of the proof. We would also want to know whether analogous results may be possible for the original jackknife.
In fact, our next result constructs explicit pathological examples to see that, without making assumptions, we cannot improve our theoretical guarantee for the jackknife+ (i.e., we cannot remove the factor of 2 appearing in Theorem 1), and no guarantee at all is possible for the jackknife. For completeness, we also construct an example to show zero coverage for the naive method, although for that method we expect to see undercoverage in practice, not just in pathological examples.
Theorem 2**.**
For any sample size , any , and any dimension , there exists a distribution on and a regression algorithm , such that the predictive coverage of the naive prediction interval (4) and the jackknife prediction interval (7) satisfy
[TABLE]
Furthermore, if , there exists a distribution on and a regression algorithm , such that the predictive coverage of jackknife+ satisfies
[TABLE]
The proof of this theorem, and the proofs for all our theoretical results presented below, are deferred to Appendix B. The example for the original jackknife is simple—we choose the regression algorithm so that models fitted at sample size are always less accurate than models fitted at sample size . The construction for jackknife+ is substantially more technical, and is similar in spirit to the example sketched in Vovk (2015, Appendix A) for cross-conformal predictors in the setting of exchangeable data. (The constant 6 on the vanishing term in the bound for jackknife+ is simply an artifact of the proof, and can certainly be improved with a more careful construction.)
2.2 The jackknife-minmax method
We have seen that the best possible coverage guarantee for jackknife+, in the assumption-free setting, is rather than the target level . To address this gap, we can consider a more conservative alternative to the jackknife+, which will remove the factor of 2 from the theoretical bound. We define the jackknife-minmax method as follows:
[TABLE]
It is simple to verify that this interval is strictly more conservative than jackknife+, meaning that for any data set, we have
[TABLE]
The advantage that jackknife-minmax provides is that, without any assumptions on the algorithm or distribution of the data, it always achieves the target coverage rate.
Theorem 3**.**
The jackknife-minmax prediction interval satisfies
[TABLE]
In practice, however, we will see that the jackknife-minmax prediction interval is generally too conservative.
3 CV+ for -fold cross-validation
Suppose that we split the training sample into disjoint subsets each of size (assumed to be an integer). Let
[TABLE]
be the regression function fitted onto the training data with the th subset removed. To assess the quality of our regression algorithm using cross-validation (CV), we would consider the residuals from this -fold process, namely,
[TABLE]
where identifies the subset that contains , i.e., . Using these residuals, we can define the CV+ prediction interval as
[TABLE]
Of course, jackknife+ can be viewed as a special case of CV+, by setting . The advantage of the CV+ method, when we choose a smaller , is that we only need to compute rather than models—however, this will likely come at the cost of slightly wider intervals, because the models are fitted using a lower sample size (i.e., ) and will lead to slightly larger residuals.
3.1 Assumption-free guarantee for CV+
Our next result verifies that the CV+ prediction interval enjoys essentially the same worst-case coverage guarantee as jackknife+.
Theorem 4**.**
The -fold CV+ prediction interval satisfies the following coverage guarantees:
- (a)
(Adapted from Vovk and Wang (2012); Vovk et al. (2018)**.)
[TABLE] 2. (b)
[TABLE]
Combining the two bounds, it follows that for all ,
[TABLE]
The first part of this result, part (a), is derived from the work Vovk and Wang (2012); Vovk et al. (2018)—we give more details on this in Section 3.2 below. This known result proves that the worst-case coverage is essentially when is sufficiently small, i.e., . Our new work, proving part (b), completes the picture by giving a meaningful bound for the case where is large (at the extreme, for leave-one-out methods). By combining the two bounds, we see that coverage is essentially at any , since the excess noncoverage is at most uniformly over any choice of .
We can also compare our result to the holdout or split conformal method (3), which is equivalent to fitting a model and constructing the prediction interval using the quantile of the residuals for , but using only a single subset (without repeating times for each fold in the partition ). As discussed earlier, this method offers an assumption-free guarantee of coverage, but this comes at the cost of higher variance due to the single split—in contrast, CV+ reduces variance by averaging over all splits, but at the cost of a weaker theoretical guarantee.
3.2 Related method: cross-conformal predictors
Our proposed CV+ prediction interval is related to the cross-conformal prediction method of Vovk (2015); Vovk et al. (2018), which (in its symmetric version) returns the predictive set
[TABLE]
Here introduces randomization into the method. By comparing to CV+, we can verify that
[TABLE]
deterministically (we demonstrate this in Appendix B.2.1). The two methods will sometimes produce the same output, but not always—in particular, may in principle return a predictive set that is a disjoint union of multiple intervals, while CV+ always returns an interval.
We next compare our theoretical findings with the known results for cross-conformal. Vovk et al. (2018) show that the -fold cross-conformal method has coverage at least222Vovk et al. (2018) do not state this coverage result directly, but instead prove coverage for a modification of the cross-conformal method; however, these two formulations can be shown to be equivalent. We give details in Appendix B.2.1.
[TABLE]
When is small, this additional term is negligible, and so we essentially have coverage for cross-conformal. However for large , such as for the leave-one-out method, their earlier result does not yield a meaningful guarantee—the guaranteed coverage level is zero. In contrast, our new assumption-free result in Theorem 1 proves coverage for the jackknife+ method (i.e., with folds), and Theorem 4 ensures coverage for -fold CV+ at any choice of .
Remark 1**.**
By examining the proofs of Theorems 1 and 4, we can see that the arguments apply directly to the -fold (or -fold) cross-conformal method; that is, our proofs for these theorems also establish that
[TABLE]
for -fold cross-conformal with any . In the special case that we are guaranteed coverage . The first term in the minimum was established by Vovk et al. (2018) as presented in (14) above, but the second term (which allows for meaningful coverage for large values of , e.g., ) is a new result.
3.3 An alternative method: conformal prediction
The final related method we present is conformal prediction (Vovk et al., 2005). (We will sometimes refer to this method as “full” conformal prediction in order to distinguish it from the split conformal or cross-conformal methods described above.) Given the base algorithm , the full conformal prediction method outputs a prediction set (which consists of a union of one or more intervals) constructed as follows:
[TABLE]
where
[TABLE]
denotes the output of the algorithm run on the training data augmented with the hypothesized test point . In other words, to determine whether to include a value in the prediction set at a new point , we need to train the algorithm on the training+test data (as though were the true response value), and then see whether the residual of the test point “conforms” with the residuals on the remaining points. The exchangeability of the test and training data ensures that
[TABLE]
i.e., coverage at the target level. However, this desirable theoretical property comes at a high cost—we can see by construction of the prediction interval that the training algorithm needs to be rerun for every test point feature vector we might consider, and for every possible response value (or in practice, for each in a fine grid over ). In certain special cases there are computational tricks allowing for efficient calculation of the prediction set—for example linear regression or ridge regression (Burnaev and Vovk, 2014), and the Lasso (Lei, 2017). Outside of these special cases, full conformal is prohibitively expensive in practice, even on moderately sized data sets; while it provides an extremely elegant and theoretically rigorous framework for distribution-free inference, it is not practical in many applied settings.
4 Summary of coverage guarantees and computational costs
In light of these theoretical results, which method should a statistician choose in practice? The following table summarizes the theoretical results behind each of the methods under consideration, and the typical empirical performance that we have observed.
Given the theoretical and empirical properties of the various options, we therefore recommend the jackknife+ as a practical alternative to the usual jackknife predictive intervals. On the one hand, the empirical performance of the jackknife+ is nearly identical to that of the original jackknife (assuming we avoid pathological examples), with both methods giving intervals of nearly the same width and achieving close to the target coverage level. However, while the jackknife offers no theoretical guarantees in the absence of stability assumptions, the jackknife+ achieves at least coverage in the worst possible case. On the other hand, the methods achieving (rather than ) coverage guarantees are either less statistically efficient in the sense of producing wider intervals (split conformal uses models fitted on a smaller portion of the data while jackknife-minmax is generally too conservative), or suffer from computational infeasibility (full conformal is computationally prohibitive aside from perhaps a few special cases).
We now turn to a direct comparison of the computational costs of these eight methods. The split conformal and naive methods require only one run of the regression algorithm (to fit on the full training data), while each of the jackknife methods requires runs (to fit for each —and one additional run to fit , in the case of the original jackknife). If the training sample size is so large that fitting regression functions is not feasible, we may instead prefer to use -fold cross-validation. In contrast, the full conformal method must train many more times—once for each possible combination of a test point feature vector and a possible response value —except for special cases such as linear regression or ridge regression. These observations are summarized below:
This table compares the computational cost (ignoring constants) of each method when run on a training sample of size , for producing prediction sets on many test points. The middle column (“Model training cost”) counts the number of times that the model fitting algorithm is run on a training data set333It is worth mentioning that for several common regression algorithms, the leave-one-out residuals can be obtained without refitting times, but by simply reweighting the in-sample training residuals. Examples include linear smoothing methods like ordinary least squares, kernel ridge regression, kernel smoothing, thin plate splines and smoothing splines. Another interesting example is random forests, where the -th leave-one-out fit can be obtained by simply ignoring all trees containing the -th point. of size (up to) . The value denotes the number of grid points of possible values (a fine grid over ), used in the construction of the full conformal prediction method (15). The last column (“Model evaluation cost”) counts the number of times we evaluate a fitted model on a single new data point. In most settings, the model training cost is dominant—for example, training a neural network is far more costly than evaluating the prediction of a trained network on a new example. There are important exceptions, however, such as -nearest neighbors, where computing a prediction incurs the cost of identifying the neighbors of the test point.
5 Guarantees under stability assumptions
Next, we consider how adding stability assumptions—conditions that ensure that the fitted regression function is not too sensitive to perturbations of the training data set—can improve the theoretical guarantees of the jackknife and its variants. (For simplicity, we only consider leave-one-out methods, and do not examine -fold cross-validation here.)
5.1 In-sample and out-of-sample stability
Fix any , , any sample size , and any distribution on . We say that a regression algorithm satisfies out-of-sample stability with respect to the distribution and sample size if, for all ,
[TABLE]
for and defined as before in (5) and (6). The probability above is taken with respect to the distribution of the data points , drawn i.i.d. from . Similarly, satisfies in-sample stability with respect to the distribution and sample size if, for all ,
[TABLE]
Naturally, since the data points are exchangeable, if (16) or (17) holds for any single then it holds for all . These types of conditions appear elsewhere in the literature—for example Bousquet and Elisseeff (2002) define similar conditions, termed “hypothesis stability” and “pointwise hypothesis stability”.
While the out-of-sample and in-sample stability properties may at first appear similar, they are extremely different in practice. Out-of-sample stability requires that, for a test point that is independent of the training data, the predicted value does not change much if we remove one point from the training data. In contrast, in-sample stability requires that, for a point in the training data set, the predicted value does not change much if we remove this point itself from the training data set. In a scenario where the model fitting algorithm suffers from strong overfitting, we would expect in-sample stability to be very poor, while out-of-sample stability may still hold—for example, we will see in Section 5.5 that this is the case for -nearest-neighbor methods. On the other hand, strongly convex regularization, such as ridge regression, induces both in- and out-of-sample stability (Bousquet and Elisseeff, 2002, Example 3). This is not the case, however, for sparse regression methods (e.g., regularization), which are proved by Xu et al. (2012) to be incompatible with in-sample stability.
5.2 Summary of stability results
Before giving the details of our theoretical results, we summarize our findings on the various methods’ predictive coverage guarantees, with and without stability assumptions:
The assumption free results are the same as those discussed in Section 4, while the results under stability assumptions are presented next in Theorems 5 and 6.
5.3 Out-of-sample stability and the jackknife
We will next prove that out-of-sample stability is sufficient for the jackknife and jackknife+ methods to achieve the target coverage rate, with a slight modification. Define
[TABLE]
and similarly, define
[TABLE]
which we refer to as the -inflated versions of the jackknife and jackknife+ predictive intervals.
Theorem 5**.**
Suppose that the regression algorithm satisfies the out-of-sample stability property (16) with respect to the data distribution and the sample size . Then the -inflated jackknife prediction interval satisfies
[TABLE]
Similarly, the -inflated jackknife+ prediction interval satisfies
[TABLE]
(The different amounts of inflation, for jackknife versus for jackknife+, are simply an artifact of the particular definition of out-of-sample stability that we use, and should not be interpreted as a meaningful difference between these two methods.)
We remark that if we additionally assume that, in the data distribution, has a bounded conditional density (for example, for some unknown true mean function ), then the result of Theorem 5 is sufficient to ensure that the (non-inflated) jackknife and jackknife+ intervals achieve close to target coverage. The reason is this: if the conditional density of is bounded by some constant , then very little probability can be captured by inflating the interval. Specifically,
[TABLE]
Combined with the result of Theorem 5, this proves that
[TABLE]
Similarly, for jackknife+, we have
[TABLE]
5.4 In-sample stability and overfitting
To contrast the scenarios of in-sample and out-of-sample stability, we will next demonstrate that adding the in-sample stability assumption would in fact be sufficient for the “naive” prediction interval, defined earlier in (4), to have coverage at roughly the target level. Its -inflated version is defined as
[TABLE]
Recall from Section 1 that we would typically expect to undercover severely due to the overfitting problem (thus inspiring the use of the jackknife to avoid this issue), and similarly will also undercover whenever is too small to correct for overfitting. This is often the case even when out-of-sample stability is satisfied. With in-sample stability, however, this is no longer the case—in other words, the in-sample stability property is essentially assuming that inflation by is sufficient to correct for overfitting.
Theorem 6**.**
Suppose that the regression algorithm satisfies both the in-sample stability property (17) and the out-of-sample stability property (16) with respect to the data distribution and the sample size . Then the naive prediction interval satisfies
[TABLE]
5.5 Example: -nearest-neighbors
To give an illustrative example, consider a -nearest-neighbor (-NN) method. This style of example is also considered in Steinberger and Leeb (2018, Example 4.1), and was studied earlier by Devroye and Wagner (1979) in the context of estimating the error of a classifier, and by Bousquet and Elisseeff (2002) in the context of error in regression. Given a training data set and a new test point , our prediction is
[TABLE]
where is the set of the nearest neighbors to the test point , i.e., the indices giving the smallest values of (of course, we can replace the norm with any other metric). We will assume for simplicity that there are no ties between these distances (for instance, the points might be continuously distributed on , or we apply a random tie-breaking rule). Now consider out-of-sample stability. Let and be the -nearest-neighbor sets for the test point given the full training data, or the training data with data point removed, respectively. Then we can easily verify that
[TABLE]
Therefore,
[TABLE]
where the last step holds by exchangeability of the training points. This proves that the -NN method satisfies -out-of-sample stability with and . (In contrast, we cannot hope for a similar argument to guarantee in-sample stability, since we will always have ; that is, is one of its own nearest neighbors—and so in general we will have .)
Applying the conclusion of Theorem 5 to this setting, then, we see that -NN leads to a coverage rate at least
[TABLE]
for the jackknife, and
[TABLE]
for the jackknife+. These results hold with no assumptions whatsoever on the distribution of the data—and in particular, we do not need to assume that the -NN prediction is accurate or consistent on the given data.
5.6 Comparison to existing results
As mentioned above, Bousquet and Elisseeff (2002) study stability in the context of generalization bounds for regression, with the aim of bounding risk rather than predictive inference. The predictive accuracy of the jackknife under assumptions of algorithm stability was explored by Steinberger and Leeb (2016) for the linear regression setting, and in a more general setting by Steinberger and Leeb (2018). Their stability assumption (see, e.g., Steinberger and Leeb (2018, Definition 1)) is essentially equivalent to our out-of-sample stability condition (16). However, the theory obtained in their work is asymptotic, and relies also on distributional assumptions (see Steinberger and Leeb (2018, (C1))), namely, that where the noise is continuously distributed and is independent of (for example, this does not allow for heteroskedasticity). In contrast, our guarantee, in Theorem 5, offers a simple finite-sample coverage guarantee with no distributional assumptions, requiring only algorithm stability.
6 Proof of Theorem 1
Suppose for a moment that we have access to the test point as well as the training data. For any indices with , let define the regression function fitted on the training plus test data, with points and removed. (We use rather than to remind ourselves that the model is fitted on a subset of the combined training and test data , rather than a subset of only the training data.) Note that for any , and that for any .
Next, we define a matrix of residuals, , with entries
[TABLE]
i.e., the off-diagonal entries represent the residual for the th point when both and are left out of the regression. We also define a comparison matrix, , with entries
[TABLE]
In other words, is the indicator for the event that, when excluding data points and from the regression, data point has higher residual than data point . Naturally we see that implies , for any . We note that this comparison matrix construction is also examined by Vovk (2015, Appendix A), where it is used to establish that leave-one-out conformal methods fail to achieve coverage.
Next, we are interested in finding data points with unusually large residuals—the ones that are hardest to predict. We will define a set of “strange” points,444The authors thank an anonymous reviewer for suggesting this presentation of the proof.
[TABLE]
where is the th row sum of . In other words, the th point is “strange” (i.e., ) if it holds that, when we compare the residual of the th point against residual for the th point (for each ), the residual for the th point is the larger one, for a sufficiently high fraction of these comparisons.
From this point on, the proof will proceed as follows:
- •
Step 1: we will establish deterministically that , that is, for any comparison matrix it is impossible to have more than many strange points.
- •
Step 2: using the fact that the data points are i.i.d. (or more generally exchangeable), we will show that the probability that the test point is strange (i.e., ) is therefore bounded by .
- •
Step 3: finally, we will verify that the jackknife+ interval can only fail to cover the test response value if is a strange point.
Step 1: bounding the number of strange points
This bound is essentially a consequence of Landau’s theorem for tournaments (Landau, 1953). For data points and , we say that data point “wins” its game against data point , if ; that is, point has a higher residual than point , under the corresponding regression . Note that each strange point can lose against at most other strange points—this is because point must win against at least points in total since it is strange, and as we have defined it, point cannot win against itself.
Let denote the number of strange points. The key realization is now that, if we think about grouping each pair of strange points by the losing point, then we see that there are at most
[TABLE]
pairs of strange points. This is because there are at most unique possibilities for the loser, and for each such loser, it can only lose against at most other strange points, as argued above. In other words, we have established
[TABLE]
and rearranging gives , as desired.
Step 2: exchangeability of the data points
We next leverage the exchangeability of the data points to show that, since there are at most strange points among a total of points, it follows that the test point has at most probability of being strange—this reasoning uses the exchangeability of the data in exactly the same way as the conformal prediction literature (Vovk et al., 2005).
To establish this formally, since the data points are exchangeable and the regression fitting algorithm is invariant to the ordering of the data points (the symmetry condition (8)), it follows that for any permutation matrix , where denotes equality in distribution. In particular, for any index , suppose we take to be any permutation matrix with (i.e., corresponding to a permutation mapping to ). Then, deterministically, we have
[TABLE]
and therefore,
[TABLE]
for all . In other words, if we compare an arbitrary training point versus the test point , these two points are equally likely to be strange, by exchangeability of the data. We can then calculate
[TABLE]
where the last step applies the result of Step 1.
Step 3: connecting to jackknife+
Finally, we need to relate the question of coverage of the jackknife+ interval, to our notion of strange points. Suppose that . This means that either
[TABLE]
which implies that for at least many indices , or otherwise
[TABLE]
which implies that for at least many indices . In either case, then, we have
[TABLE]
and therefore , that is, point is strange. Combining this with the result of Step 2, we have
[TABLE]
7 Empirical results
In this section, we compare seven methods—naive (4), jackknife (7), jackknife+ (9), jackknife-minmax (10), CV+ (11), split conformal (3), and full conformal (15)—on simulated and real data. Code for reproducing all results and figures is available online.555http://www.stat.uchicago.edu/~rina/jackknife.html
7.1 Simulations
We first examine the performance of the various prediction intervals on a simulated example, using least squares as our regression method. We will see that when the training sample size is equal or approximately equal to the dimension , the instability of the least squares method (due to poor conditioning of the design matrix) leads to a wide disparity in performance between the various methods. This simulation is thus designed to demonstrate the role of stability in the performance of these various methods.
7.1.1 Data and methods
Our target coverage level is . We use training sample size , and repeat the experiment at each dimension , with i.i.d. data points generated as
[TABLE]
The true coefficient vector is drawn as for a uniform random unit vector . The regression method is simply least squares, with the convention that if the linear system is underdetermined then we take the solution with the lowest norm (the limit of ridge regression as the regularization tends to zero). Specifically, for training data , we return the regression function ,
[TABLE]
where denotes the matrix of covariates, the vector of responses, and the Moore–Penrose pseudoinverse.
We then generate 100 test data points from the same distribution, and calculate the empirical probability of coverage (i.e., the proportion of test points for which the prediction interval computed at the value contains the value) and the average width of the prediction interval.
7.1.2 Results
Figure 2 displays the results of the simulation, averaged over 50 trials (where each trial has an independent draw of the training sample of and the test sample of size 100).
When , the jackknife and jackknife+ show very similar performance, with approximately the right coverage level and with nearly identical interval width. For (the regime where least squares is quite unstable), the jackknife has substantial undercoverage—at the jackknife shows coverage rate around , and continues to show substantial undercoverage when is slightly larger than . In this regime, the jackknife+ continues to show the right coverage level, at the cost of a prediction interval that is only slightly wider than the jackknife. For large , the jackknife and jackknife+ again show very similar performance. In fact, this connects to recent work on interpolation methods (methods that achieve zero training error). Specifically, Hastie et al. (2019) study “ridgeless” regression (i.e., the least squares solution with the lowest norm, as in our simulation), and demonstrate that this provides a stable solution with good test error as long as is either sufficiently small or sufficiently large relative to . We see a similar phenomenon in the predictive coverage performance of the jackknife.
As expected, the jackknife-minmax is over-conservative, with typical coverage higher than across all dimensions , while the naive method drastically undercovers due to increasing overfitting as grows (and in fact, at , the training error is exactly zero, so the prediction intervals have width zero and coverage zero.)
When , we note that full conformal prediction will always have infinite length intervals since for every potential in the pair, all residuals will equal zero. Naturally, in such a situation, full conformal will have coverage equal to one deterministically. In practice, it is common to modify the conformal prediction method by truncating to a finite range, e.g., to the observed range of values in the training data (which has minimal effect on the coverage guarantee (Chen et al., 2018)); this is why we see finite length intervals for full conformal in our simulation results.
Split conformal is the only method other than jackknife+ to maintain coverage at throughout. (Note that since split conformal trains on half of the data, its length spikes near , rather than as for the other methods; this is simply due to the change in sample size used in training. This is a result of instability of OLS when and is not reflective of comparisons between holdout and jackknife+.)
7.2 Real data
We next compare the various methods on three real data sets. We will try three regression algorithms: ridge regression, random forests, and neural networks (details given below). Our aim in these experiments is to demonstrate the typical performance of the various prediction interval methods in a real data setting; we do not seek to optimize the base methods used as our regression algorithms, but are only interested in how the various prediction interval methods behave in comparison to each other. Due to the high computational cost of the full conformal method, we do not include it in the comparison.
7.2.1 Data
The Communities and Crime data set666http://archive.ics.uci.edu/ml/datasets/communities+and+crime (Redmond and Baveja, 2002) contains information on 1994 communities, with covariates such as median income, distribution of ages, family size, etc., and the goal of predicting a response variable defined as the per capita violent crime rate. After removing categorical variables and variables with missing data, covariates remain.
The BlogFeedback data set777https://archive.ics.uci.edu/ml/datasets/BlogFeedback (Buza, 2014) contains 52397 data points, each corresponding to a single blog post. The goal is to predict the response variable of the number of comments left on the blog post in the following 24 hours, using covariates such as the length of the post, the number of comments on previous posts, etc. Since the distribution of the response is extremely skewed, we transform it as .
The Medical Expenditure Panel Survey 2016 data set,888https://meps.ahrq.gov/mepsweb/data_stats/download_data_files_detail.jsp?cboPufNumber=HC-192 provided by the Agency for Healthcare Research and Quality, contains data on individuals’ utilization of medical services such as visits to the doctor, hospital stays, etc. Details on the data collection for older versions of this data set are described in Ezzati-Rice et al. (2008). We select a subset of relevant features, such as age, race/ethnicity, family income, occupation type, etc. After splitting categorical features into dummy variables to encode each category separately, the resulting dimension is . The goal is to predict the health care system utilization of each individual, which is a composite score reflecting the number of visits to a doctor’s office, hospital visits, days in nursing home care, etc. With missing data removed, this data set contains 33005 data points. Since the distribution of the response is highly skewed, we transform it as .
7.2.2 Methods
Our procedure is the same for each of the three data sets. We randomly sample data points from the full data set, to use as the training data. The remaining points form the test set.
We run our experiment using three different regression algorithms —namely, ridge regression, random forests, and neural networks. The details of these algorithms is as follows:
- •
For ridge regression, we define for
[TABLE]
where the penalty parameter is chosen as , where is the covariate matrix of the training data, and is its spectral norm. This choice is to accommodate situations in which the matrix does not have full column rank as in the case where . In such cases, the solution above is nearly the least-squares solution with minimum norm.
- •
For random forests, we use the RandomForestRegressor method from the scikit-learn package (Pedregosa et al., 2011) in Python, with 20 trees grown for each random forest using the mean absolute error criterion, and with default settings otherwise.
- •
For neural networks, we use the MLPRegressor method also from scikit-learn, run with the L-BFGS solver and the logistic activation function, and with default settings otherwise.
For each choice of , we construct six prediction intervals (naive, jackknife, jackknife+, jackknife-minmax, CV+, split conformal), and calculate their empirical coverage rate and their average width on the test set. We then repeat this procedure 20 times, with the train/test split formed randomly each time, and report the mean and standard error over these 20 trials.
7.2.3 Results
Figure 3 displays the results of the real data experiments. For each data set, each regression algorithm, and each one of the six prediction interval methods, the figure plots the average coverage and average width, together with their standard errors across the 20 independent trials.
We see that the jackknife and jackknife+ methods both yield empirical coverage extremely close to the target level of 90%, and have very similar predictive interval widths. However, in some settings, the jackknife+ shows slightly higher coverage than jackknife, and slightly wider prediction intervals. These settings correspond to regression methods with greater instability. As expected, the naive method undercovers in some settings and the jackknife-minmax is generally overly conservative. Split conformal performs reasonably well: its length and coverage is sometimes comparable to the jackknife+, but is also significantly wider in some instances. Intuitively, if the best regression function in the considered function class is simple and the dataset is large, split conformal should perform fine even though it uses points for training and for calibration; however, in settings where the dataset is small relative to the complexity of the best regressor, then we should observe significant gains in using points for training and points for calibration. One phenomenon that is not visible in the empirical results is that split conformal is a randomized method, with output varying slightly depending on the random split, while jackknife+ is a deterministic method on any fixed training data set.
8 Summary
The jackknife+ differs from the jackknife in that it uses the quantiles of
[TABLE]
instead of those of , to build predictive intervals. By applying the shifts , the jackknife+ effectively accounts for the (possible) algorithm instability, yielding rigorous coverage guarantees under no assumptions other than exchangeable samples. This, together with its empirical performance on real data, makes it a better choice than the jackknife in practice. In cases where the jackknife+ is computationally prohibitive, -fold CV+ offers an attractive alternative. Here, it would be interesting to see if the coverage guarantees for the latter method can be somewhat sharpened.
Acknowledgements
The authors are grateful to the American Institute of Mathematics for supporting and hosting our collaboration. R.F.B. was partially supported by the National Science Foundation via grant DMS–1654076 and by an Alfred P. Sloan fellowship. E.J.C. was partially supported by the Office of Naval Research under grant N00014-16-1-2712, by the National Science Foundation via grant DMS–1712800, and by a generous gift from TwoSigma. The authors are grateful to an anonymous reviewer for helpful suggestions on the presentation of the proof of Theorem 1. E.J.C. thanks Yaniv Romano for help with some experiments. A.R. thanks Arun Kumar Kuchibhotla for discussions regarding cross-conformal prediction.
Appendix A Asymmetric jackknife+ and CV+
In settings where the distribution of given appears to have symmetric noise, it is natural to construct predictions intervals symmetrically, which is why we can consider absolute values of residuals. If the data is likely to be skewed, however, we may want to consider an asymmetric construction. Fix any with , and let
[TABLE]
where the signed residuals are
[TABLE]
We can of course consider the analogous asymmetric version of the original jackknife,
[TABLE]
This type of asymmetric jackknife was considered by Steinberger and Leeb [2018]. Similarly we can define an asymmetric version of jackknife-minmax or of CV+. We remark that, even if we were to choose , these asymmetric constructions would not necessarily be equal to the original jackknife, jackknife+, jackknife-minmax, and CV+ intervals, because the empirical distribution of the signed residuals will in general be asymmetric even if only due to random chance.
All of the coverage guarantees that we have proved for the various symmetric methods, hold also for their asymmetric counterparts. For example, to verify coverage for the asymmetric jackknife+ in the assumption-free setting, the proof of Theorem 1 proceeds identically except that the matrix of residuals constructed in the proof is replaced with two matrices
[TABLE]
where (resp. ) is used to bound the probability of noncoverage in the right (resp. left) tail by (resp. ).
Appendix B Additional proofs
B.1 Proof of jackknife-minmax (Theorem 3)
The proof for jackknife-minmax proceeds nearly identically to the proof for jackknife+ (Theorem 1). We define the residuals exactly as in the proof of Theorem 1, but we will use a different definition for the matrix :
[TABLE]
that is, the smallest residual for data point (leaving out any point ) is larger than , which is data point ’s residual when leaving out point . Define exactly as before. Now we follow essentially the same three steps as in the proof of Theorem 1:
- •
Step 1: we will establish deterministically that, for this new definition of the matrix , we have (whereas, for jackknife+, the bound was ).
- •
Step 2: using the fact that the data points are i.i.d. (or more generally exchangeable), we will show that the probability that the test point is strange (i.e., ) is therefore bounded by .
- •
Step 3: finally, we will verify that the jackknife-minmax interval can only fail to cover the test response value if is a strange point.
Step 1: bounding the number of strange points
To prove Step 1, let
[TABLE]
Then by definition, for all , . This means that, by definition of the new comparison matrix , we have for all , and therefore,
[TABLE]
where the last step holds by definition of (since is a strange point). Therefore as desired.
Step 2: exchangeability of the data points
This step is identical to Step 2 in the proof of Theorem 1.
Step 3: connecting to jackknife-minmax
Suppose that . This means that either
[TABLE]
which implies that for at least many indices , or otherwise
[TABLE]
which implies that for at least many indices . In either case, then, we have
[TABLE]
and therefore .
B.2 Proofs for the CV+ method
In this section, we will give details for how the CV+ method relates to the cross-conformal prediction method, and then prove our theoretical guarantees for CV+.
B.2.1 Details for comparing to the cross-conformal method
In Section 3.2, we introduced the cross-conformal method (12) of Vovk [2015], Vovk et al. [2018], and stated two properties—first, that the cross-conformal prediction set is always contained in the CV+ prediction interval (13), and second, that the results of Vovk et al. [2018] imply a coverage guarantee (14) for the cross-conformal method. Here we give details to justify these two statements.
First, we verify that the cross-conformal prediction set is contained in the CV+ interval. To see this, suppose that . Then by definition of this predictive set, we have
[TABLE]
for some . Since the left-hand side is monotone in , in particular this implies that the above inequality holds at , and so
[TABLE]
Therefore,
[TABLE]
meaning that is not larger than the -th smallest value of , . In other words,
[TABLE]
An identical argument proves that
[TABLE]
meaning that must lie in the CV+ prediction interval . This completes our verification of the claim (13) that the CV+ interval contains the cross-conformal prediction set deterministically.
Next we give details for the coverage guarantee for the cross-conformal method, which is implied but not stated explicitly by Vovk et al. [2018]. Specifically, Vovk et al. [2018] show that a modification of the -fold cross-conformal method can lead to a coverage guarantee. To define the modified method, let
[TABLE]
where is the number of data points in each fold (assumed to be an integer). Plugging in the true test value , we see that is a rank-based -value comparing the test residual \big{|}Y_{n+1}-\widehat{\mu}_{-S_{k}}(X_{n+1})\big{|} of the test point against all other residuals in the th fold. (Here corrects for discretization, so that this p-value is uniformly distributed on instead of on a discrete grid.) Vovk et al. [2018] then consider a modified cross-conformal method,
[TABLE]
Vovk et al. [2018] cite earlier work by Vovk and Wang [2012, Corollary 2], which proves that an arithmetic means of -values is itself a valid p-value up to a factor of 2—that is,
[TABLE]
for any , and so the modified cross-conformal method at level has predictive coverage at least .
Now we relate the modified cross-conformal method to its original version. Plugging in the definition of the p-values and comparing with the original (unmodified) cross-conformal predictive set (12), we can see that, deterministically,
[TABLE]
Since the modified cross-conformal method, run at level , has coverage at least , this proves that
[TABLE]
B.2.2 Proof of Theorem 4
This proof follows essentially the same steps as the proof of Theorem 1. Suppose that we draw additional test points, so that in total we have many test points, , . After partitioning the training data into sets of size , we define , the set of test points.
For any with , let define the regression function fitted on the training plus test data, with subsets and removed. Next, we define a matrix of residuals, with entries
[TABLE]
Define a comparison matrix with
[TABLE]
and consider the set of “strange” points,
[TABLE]
where . Now we proceed as for the proof of Theorem 1.
- •
First, for Step 1, we bound the number of strange points deterministically as .
- •
For Step 2 we see that the exchangeability of the data points implies that the probability that the test point is strange (i.e., ) is therefore bounded by .
- •
For Step 3, we see that noncoverage of the CV+ interval implies that the test point is strange, which completes the proof of the theorem.
For Step 3, the proof that implies that , is identical to the corresponding step in Theorem 1. We remark that in fact, our argument verifies a strictly stronger statement: if is not contained in the -fold cross-conformal prediction set (with , in the case of jackknife+), then the test point is strange—this is strictly stronger because the cross-conformal prediction set satisfies always.
Turning to Step 1, we now bound the number of strange points. This step is very similar to the proof of Step 1 in the proof of Theorem 1. Let be the number of strange points in the th fold, so that . As before, each strange point can lose against at most other strange points. However, the difference relative to jackknife+ is that the data points and in the same fold () do not play against each other in the “tournament” so there are two types of pairs of strange points: those that are in different folds (writing , there are at most s\cdot\big{(}\alpha(n+1)-1\big{)} such pairs, as in the jackknife+ proof), and those that are in the same fold (and therefore do not play a game). Thus we have established that
[TABLE]
We can simplify the last sum as
[TABLE]
since for each . Simplifying the expression above we have therefore proved that
[TABLE]
as desired.
Finally, we verify Step 2. For the -fold setting, this is a bit more subtle than for jackknife+. This is because we cannot claim that for any permutation matrix —indices belonging in the same fold () behave differently than indices in different folds (). However, treating the split of the training and test data into folds as fixed, we can verify that for any permutation matrix that preserves the equivalence relation induced by the folds, if . Now, for any , there exists such a with . Thus, as in the proof of Theorem 1, this implies that for all . Therefore, combining with the result of Step 1,
[TABLE]
Combining with Step 3, we have completed the proof of the coverage guarantee.
B.3 Proof of Theorem 5
We will first consider an oracle leave-one-out method that, while impossible to implement in practice, achieves the target coverage rate. We will then relate the -inflated jackknife and -inflated jackknife+ to the oracle method.
B.3.1 Oracle method
For each , let be the regression function fitted on the training and test data with point removed, i.e.
[TABLE]
Note that , while for , differs from since the test point is included in the regression. Consider an “oracle jackknife” method, where we use in place of :
[TABLE]
where and
[TABLE]
Next, we will confirm that this oracle method achieves coverage. This fact is based on the exchangeability of the training and test data points, and can be proved using standard techniques from the conformal prediction literature (see, e.g., Vovk et al. [2005], Lei et al. [2018] for background). Writing
[TABLE]
we see that failure to cover, i.e., , occurs if and only if
[TABLE]
Now, since the training and test data are i.i.d., and the algorithm is assumed to be invariant to the labeling of the points (8), this means that the resulting oracle residuals are exchangeable. In other words, the rank of among this list of oracle residuals is uniformly random. Therefore, the probability that the event in (22) occurs is equal to the probability that is not one of the smallest values in the list of oracle residuals, and so
[TABLE]
(The first inequality cannot be replaced with an equality, due to the possibility of ties among the residuals.)
B.3.2 Bound for the jackknife
Now we relate the oracle method back to the jackknife. We will show that
[TABLE]
with sufficiently high probability. To see why, suppose instead that this set inclusion does not hold. Then by definition of , we must have
[TABLE]
By definition of these quantiles, we can conclude that the number of indices with is at least
[TABLE]
Therefore,
[TABLE]
where the last step holds by Markov’s inequality. Observe also that, for every ,
[TABLE]
where the first and third step hold by definition of the various regression functions; the second step holds since the data points are i.i.d. and so swapping the label of point and point does not change the distribution; and the last step holds by applying out-of-sample stability (16). Therefore, returning to (24), we have
[TABLE]
Combining this bound with (23), we have therefore proved that
[TABLE]
B.3.3 Bound for the jackknife+
The argument for the jackknife+ proceeds similarly, except that we now compare against the jackknife rather than the oracle—we will verify that
[TABLE]
holds with sufficiently high probability, where we again define .
Suppose that this does not hold. Then it must either fail at the upper bounds of the intervals, i.e.,
[TABLE]
or at the lower bounds, i.e.,
[TABLE]
In the first case, this implies that for at least many indices , while in the second case, we instead have for at least many indices . Combining the two, then, we have
[TABLE]
where we again apply Markov’s inequality and the out-of-sample stability property just as for the jackknife proof. Combining the coverage result (25) (with in place of ) with (26), we have proved that
[TABLE]
B.4 Proof of Theorem 6
Let R^{\textnormal{naive}}_{i}=\big{|}Y_{i}-\widehat{\mu}(X_{i})\big{|} be the th residual in the “naive” method while R^{\textnormal{LOO}}_{i}=\big{|}Y_{i}-\widehat{\mu}_{-i}(X_{i})\big{|} is the leave-one-out residual as before. Suppose that we run jackknife at level , where . Then by definition of the two methods, we have
[TABLE]
and
[TABLE]
We will now check that
[TABLE]
with sufficiently high probability. Following similar arguments as in the proof of Theorem 5, if this does not hold, then it must be the case that
[TABLE]
By the triangle inequality, this implies that
[TABLE]
Therefore, by Markov’s inequality,
[TABLE]
where the second step applies the in-sample stability property (17). Therefore,
[TABLE]
where for the next-to-last step we apply the result of Theorem 5 (with in place of ).
B.5 Proof of Theorem 2
In this section, we will prove a stronger version of Theorem 2, and will verify that the lower bounds on coverage hold even when we use the -inflated versions of each of the intervals, for any . That is, we will construct pathological examples for which, for the naive method and for jackknife, we have
[TABLE]
and for jackknife+ with , we have
[TABLE]
B.5.1 Proof for jackknife and naive methods
First we construct a simple example for the jackknife and naive intervals. Define the regression method as follows: given a training sample of size , the algorithm returns the function
[TABLE]
Now we define the data distribution on : let , and let . Then with probability 1, the values will be distinct. In this case, the leave-one-out residuals will be given by
[TABLE]
for all , yielding a residual quantile \widehat{q}^{\,{}_{+}}_{n,\alpha}\big{\{}R^{\textnormal{LOO}}_{i}\big{\}}=(1+\epsilon)(n-1) and a prediction interval
[TABLE]
Therefore, the -inflated interval is given by
[TABLE]
However, with probability 1, and so
[TABLE]
Similarly, for the naive method, its residuals will be given by
[TABLE]
and so following the same argument we see that
[TABLE]
A simple calculation shows that in this example, jackknife+ interval contains 0 at its left endpoint, and hence maintains its coverage.
B.5.2 Proof for jackknife+
Next we give the construction for the jackknife+. Our construction is similar in spirit to the example constructed by Vovk [2015, Appendix A], which gives intuition for why the leave-one-out cross-conformal predictor (described earlier in Section 2) may fail when the data points are only assumed to be exchangeable (specifically, the data points and their residuals are chosen deterministically, and then randomly permuted). Here our construction is more technical as we need to work in the setting of i.i.d. data.
To give intuition we first sketch the idea. Suppose that the distribution of is chosen such that, with probability , is drawn from some “bad” region where predicting is challenging and we consistently underestimate , while with the remaining probability, is drawn from some “good” region where predicting is easy—in fact, we can do this with zero error. Now, what is the chance that is not covered by the jackknife+? If is “good”, then will be covered. If is “bad”, then we will have
- •
For approximately half of the “bad” ( data points), we will have
[TABLE]
- •
For all “good” and for the remaining “bad” (in total, of the data points), we will have
[TABLE]
This will be sufficient to see that is almost certainly not covered by the jackknife+ interval whenever is “bad”, i.e., with probability .
Now we give the formal construction. Fix a small and a large , which we will specify later on. First, we will choose the distribution for the data. Let999To obtain this distribution, if the dimension is we can take the above distribution over the first 3 coefficients in and simply ignore the remaining coefficients, while if or we can transform the data if needed; for example if , taking , we can use the first digits of to generate a draw where p\approx\big{(}2\alpha(1-\gamma)\big{)} (up to accuracy), then we use the next digit of to draw , and finally let be defined by the remaining digits of .
[TABLE]
and let . Next we define the regression algorithm as follows: given a training sample (X_{j},Y_{j})=\big{(}(A_{j},B_{j},C_{j}),Y_{j}\big{)} indexed over , the resulting fitted regression function is defined as
[TABLE]
at any point . Defining , we therefore have
[TABLE]
for each . Now we check that coverage is roughly . We have
[TABLE]
Next we verify that this last probability is close to 1. These indicator variables are independent conditional on (as they then depend only on for each ). We denote the conditional probabilities by
[TABLE]
and calculate
[TABLE]
By Hoeffding’s inequality, we have
[TABLE]
for any . Choosing and marginalizing, we have
[TABLE]
Combining everything so far, we have therefore proved that
[TABLE]
Now we bound this last probability. We have
[TABLE]
where the last step holds since A_{i}B_{-i}C_{n+1}=A_{i}B_{i}\cdot\big{(}C_{n+1}\prod_{j=1}^{n}B_{j}\big{)}. Next, holds with probability , while by Hoeffding’s inequality,
[TABLE]
and
[TABLE]
Putting these calculations together,
[TABLE]
After simplifying,
[TABLE]
Now we choose and (we can assume that , since otherwise the theorem is trivial as it only claims that the coverage rate is no higher than 1). With this choice, we calculate
[TABLE]
and so returning to our earlier calculations we have
[TABLE]
thus proving the theorem.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1Bousquet and Elisseeff [2002] Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of machine learning research , 2(Mar):499–526, 2002.
- 2Burnaev and Vovk [2014] Evgeny Burnaev and Vladimir Vovk. Efficiency of conformalized ridge regression. In Conference on Learning Theory , pages 605–622, 2014.
- 3Butler and Rothman [1980] Ronald Butler and Edward D Rothman. Predictive intervals based on reuse of the sample. Journal of the American Statistical Association , 75(372):881–889, 1980.
- 4Buza [2014] Krisztian Buza. Feedback prediction for blogs. In Data analysis, machine learning and knowledge discovery , pages 145–152. Springer, 2014.
- 5Chen et al. [2018] Wenyu Chen, Kelli-Jean Chun, and Rina Foygel Barber. Discretized conformal prediction for efficient distribution-free inference. Stat , 7(1):e 173, 2018.
- 6Devroye and Wagner [1979] Luc Devroye and Terry Wagner. Distribution-free inequalities for the deleted and holdout error estimates. IEEE Transactions on Information Theory , 25(2):202–207, 1979.
- 7Efron [1979] Bradley Efron. Bootstrap methods: Another look at the jackknife. Ann. Statist. , 7(1):1–26, 1979.
- 8Efron and Gong [1983] Bradley Efron and Gail Gong. A leisurely look at the bootstrap, the jackknife, and cross-validation. The American Statistician , 37(1):36–48, 1983.