Optimal Implicit Bias in Linear Regression

TL;DR

This paper analyzes the implicit biases in over-parameterized linear regression to identify the conditions and biases that lead to optimal generalization performance, providing a theoretical foundation for understanding and improving over-parameterized models.

Contribution

It offers a precise asymptotic analysis of the generalization error for interpolators from convex minimization, identifying the optimal implicit bias for best generalization in over-parameterized linear regression.

Findings

Derived a tight lower bound on the best possible generalization error.

Identified the optimal convex implicit bias achieving this lower bound.

Provided conditions involving data distribution and prior for optimal bias.

Abstract

Most modern learning problems are over-parameterized, where the number of learnable parameters is much greater than the number of training data points. In this over-parameterized regime, the training loss typically has infinitely many global optima that completely interpolate the data with varying generalization performance. The particular global optimum we converge to depends on the implicit bias of the optimization algorithm. The question we address in this paper is, ``What is the implicit bias that leads to the best generalization performance?". To find the optimal implicit bias, we provide a precise asymptotic analysis of the generalization performance of interpolators obtained from the minimization of convex functions/potentials for over-parameterized linear regression with non-isotropic Gaussian data. In particular, we obtain a tight lower bound on the best generalization error possible among this class of interpolators in terms of the over-parameterization ratio, the variance of the noise in the labels, the eigenspectrum of the data covariance, and the underlying distribution of the parameter to be estimated. Finally, we find the optimal convex implicit bias that achieves this lower bound under certain sufficient conditions involving the log-concavity of the distribution of a Gaussian convolved with the prior of the true underlying parameter.