A New Formula for Faster Computation of the K-Fold Cross-Validation and Good Regularisation Parameter Values in Ridge Regression

TL;DR

This paper introduces a new theorem that enables faster and exact computation of k-fold cross-validation residuals in ridge regression without refitting models, improving efficiency especially for small group segments.

Contribution

The authors present a novel update formula for residuals in k-fold cross-validation that eliminates the need for repeated model fitting, applicable to various strategies and including special cases like leave-one-out.

Findings

Significant reduction in computation time for cross-validation residuals.

Effective heuristic for fast approximation in small group segments.

Demonstrated improvements in parameter selection for ridge regression.

Abstract

In the present paper, we prove a new theorem, resulting in an update formula for linear regression model residuals calculating the exact k-fold cross-validation residuals for any choice of cross-validation strategy without model refitting. The required matrix inversions are limited by the cross-validation segment sizes and can be executed with high efficiency in parallel. The well-known formula for leave-one-out cross-validation follows as a special case of the theorem. In situations where the cross-validation segments consist of small groups of repeated measurements, we suggest a heuristic strategy for fast serial approximations of the cross-validated residuals and associated Predicted Residual Sum of Squares (PRESS) statistic. We also suggest strategies for efficient estimation of the minimum PRESS value and full PRESS function over a selected interval of regularisation values. The computational effectiveness of the parameter selection for Ridge- and Tikhonov regression modelling resulting from our theoretical findings and heuristic arguments is demonstrated in several applications with real and highly multivariate datasets.