Muddling Labels for Regularization, a novel approach to generalization

TL;DR

This paper introduces a new regularization approach that directly measures and minimizes overfitting risk, enabling hyperparameter calibration during training without data splitting, and shows improved generalization in linear regression tasks.

Contribution

A novel risk measure for regularization that allows hyperparameter tuning during training without validation data, applicable to various structures and compatible with gradient descent.

Findings

Procedures outperform traditional cross-validation methods in generalization.

Methods are computationally feasible and easy to implement.

Approach improves estimation and support recovery of model parameters.

Abstract

Generalization is a central problem in Machine Learning. Indeed most prediction methods require careful calibration of hyperparameters usually carried out on a hold-out \textit{validation} dataset to achieve generalization. The main goal of this paper is to introduce a novel approach to achieve generalization without any data splitting, which is based on a new risk measure which directly quantifies a model's tendency to overfit. To fully understand the intuition and advantages of this new approach, we illustrate it in the simple linear regression model ($Y=X\beta+\xi$) where we develop a new criterion. We highlight how this criterion is a good proxy for the true generalization risk. Next, we derive different procedures which tackle several structures simultaneously (correlation, sparsity,...). Noticeably, these procedures \textbf{concomitantly} train the model and calibrate the hyperparameters. In addition, these procedures can be implemented via classical gradient descent methods when the criterion is differentiable w.r.t. the hyperparameters. Our numerical experiments reveal that our procedures are computationally feasible and compare favorably to the popular approach (Ridge, LASSO and Elastic-Net combined with grid-search cross-validation) in term of generalization. They also outperform the baseline on two additional tasks: estimation and support recovery of $\beta$. Moreover, our procedures do not require any expertise for the calibration of the initial parameters which remain the same for all the datasets we experimented on.