Train on Validation: Squeezing the Data Lemon

TL;DR

This paper introduces a practical method to utilize validation data for training, enabling a controlled trade-off between performance and overfitting, supported by theoretical stability analysis and empirical results on MNIST and CIFAR-10.

Contribution

It proposes a novel approach for using validation data in training, along with the concept of validation stability and theoretical guarantees linking stability to overfitting prevention.

Findings

Significant test performance improvement on MNIST and CIFAR-10.

Minor bias increase in model selection process.

Validation stability ensures overfitting control.

Abstract

Model selection on validation data is an essential step in machine learning. While the mixing of data between training and validation is considered taboo, practitioners often violate it to increase performance. Here, we offer a simple, practical method for using the validation set for training, which allows for a continuous, controlled trade-off between performance and overfitting of model selection. We define the notion of on-average-validation-stable algorithms as one in which using small portions of validation data for training does not overfit the model selection process. We then prove that stable algorithms are also validation stable. Finally, we demonstrate our method on the MNIST and CIFAR-10 datasets using stable algorithms as well as state-of-the-art neural networks. Our results show significant increase in test performance with a minor trade-off in bias admitted to the model selection process.