Irredundant $k$-Fold Cross-Validation

TL;DR

Irredundant $k$-fold cross-validation is a new method that uses each dataset instance exactly once for training and testing, reducing redundancy, overfitting, and computational costs while maintaining reliable performance estimates.

Contribution

The paper introduces a novel irredundant $k$-fold cross-validation method that ensures each instance is used exactly once for training and testing, improving dataset utilization and analysis accuracy.

Findings

Provides consistent performance estimates across datasets

Reduces variance in model evaluation

Lowers computational costs compared to traditional methods

Abstract

In traditional k-fold cross-validation, each instance is used ($k-1$) times for training and once for testing, leading to redundancy that lets many instances disproportionately influence the learning phase. We introduce Irredundant $k$-fold cross-validation, a novel method that guarantees each instance is used exactly once for training and once for testing across the entire validation procedure. This approach ensures a more balanced utilization of the dataset, mitigates overfitting due to instance repetition, and enables sharper distinctions in comparative model analysis. The method preserves stratification and remains model-agnostic, i.e., compatible with any classifier. Experimental results demonstrate that it delivers consistent performance estimates across diverse datasets -- comparable to $k$-fold cross-validation -- while providing less optimistic variance estimates because training partitions are non-overlapping, and significantly reducing the overall computational cost.