Fast and Informative Model Selection using Learning Curve Cross-Validation

TL;DR

This paper introduces LCCV, a learning curve-based cross-validation method that speeds up model selection and provides insights into the learning process, achieving similar accuracy to traditional CV with less computation.

Contribution

The paper presents a novel validation approach using learning curves that reduces runtime and offers deeper insights into model learning, outperforming traditional CV in efficiency.

Findings

LCCV achieves similar performance to 5/10-fold CV in over 90% of cases.

LCCV reduces runtime by over 20% on average.

Provides insights into data acquisition benefits.

Abstract

Common cross-validation (CV) methods like k-fold cross-validation or Monte-Carlo cross-validation estimate the predictive performance of a learner by repeatedly training it on a large portion of the given data and testing on the remaining data. These techniques have two major drawbacks. First, they can be unnecessarily slow on large datasets. Second, beyond an estimation of the final performance, they give almost no insights into the learning process of the validated algorithm. In this paper, we present a new approach for validation based on learning curves (LCCV). Instead of creating train-test splits with a large portion of training data, LCCV iteratively increases the number of instances used for training. In the context of model selection, it discards models that are very unlikely to become competitive. We run a large scale experiment on the 67 datasets from the AutoML benchmark and empirically show that in over 90% of the cases using LCCV leads to similar performance (at most 1.5% difference) as using 5/10-fold CV. However, it yields substantial runtime reductions of over 20% on average. Additionally, it provides important insights, which for example allow assessing the benefits of acquiring more data. These results are orthogonal to other advances in the field of AutoML.