Cross Validation for Correlated Data in Regression and Classification Models, with Applications to Deep Learning

TL;DR

This paper introduces a bias-corrected cross-validation method for correlated data in regression and classification, applicable to deep learning, improving model evaluation accuracy in complex data scenarios.

Contribution

It extends bias correction in cross-validation to any model and performance criterion, addressing correlated data issues beyond linear models.

Findings

Bias-corrected CV outperforms standard CV in correlated data scenarios.

Method applicable to deep neural networks and various prediction criteria.

Validated on synthetic and real-world datasets with complex correlation structures.

Abstract

We present a methodology for model evaluation and selection where the sampling mechanism violates the i.i.d. assumption. Our methodology involves a formulation of the bias between the standard Cross-Validation (CV) estimator and the mean generalization error, denoted by $w_{cv}$, and practical data-based procedures to estimate this term. This concept was introduced in the literature only in the context of a linear model with squared error loss as the criterion for prediction performance. Our proposed bias-corrected CV estimator, $\text{CV}_c=\text{CV}+w_{cv}$, can be applied to any learning model, including deep neural networks, and to a wide class of criteria for prediction performance in regression and classification tasks. We demonstrate the applicability of the proposed methodology in various scenarios where the data contains complex correlation structures (such as clustered and spatial relationships) with synthetic data and real-world datasets, providing evidence that the estimator $\text{CV}_c$ is better than the standard CV estimator. This paper is an expanded version of our published conference paper.