The Relative Instability of Model Comparison with Cross-validation

TL;DR

This paper demonstrates that cross-validation can produce unstable and invalid model comparisons even when individual models are stable, emphasizing the need to verify stability before inference.

Contribution

It proves that simple, stable models like Lasso can lead to unstable comparisons and invalid CV inferences, challenging assumptions about CV reliability.

Findings

Lasso and soft-thresholding generate unstable comparisons

CV inferences can be invalid even with stable models

Highlights importance of checking stability before CV

Abstract

Cross-validation (CV) is known to provide asymptotically exact tests and confidence intervals for model improvement but only when the model comparison is relatively stable. Surprisingly, we prove that even simple, individually stable models can generate relatively unstable comparisons, calling into question the validity of CV inference. Specifically, we show that the Lasso and its close cousin, soft-thresholding, generate relatively unstable comparisons and invalid CV inferences, even in the most favorable of learning settings and when both models are individually stable. These findings highlight the importance of verifying relative stability before deploying CV for model comparison.