On Generalization of Adaptive Methods for Over-parameterized Linear Regression

TL;DR

This paper investigates how adaptive optimization methods perform in over-parameterized linear regression, revealing two classes with distinct generalization behaviors and supporting findings through experiments on linear models and neural networks.

Contribution

It characterizes the generalization properties of adaptive methods in over-parameterized linear regression, distinguishing two classes based on their convergence and saturation behaviors.

Findings

Adaptive methods either stay in data span and converge to minimum norm solution or have out-of-span saturation.

Experiments confirm theoretical distinctions in linear regression and neural networks.

Adaptive methods' behavior depends on their effect on parameter span and convergence properties.

Abstract

Over-parameterization and adaptive methods have played a crucial role in the success of deep learning in the last decade. The widespread use of over-parameterization has forced us to rethink generalization by bringing forth new phenomena, such as implicit regularization of optimization algorithms and double descent with training progression. A series of recent works have started to shed light on these areas in the quest to understand -- why do neural networks generalize well? The setting of over-parameterized linear regression has provided key insights into understanding this mysterious behavior of neural networks. In this paper, we aim to characterize the performance of adaptive methods in the over-parameterized linear regression setting. First, we focus on two sub-classes of adaptive methods depending on their generalization performance. For the first class of adaptive methods, the parameter vector remains in the span of the data and converges to the minimum norm solution like gradient descent (GD). On the other hand, for the second class of adaptive methods, the gradient rotation caused by the pre-conditioner matrix results in an in-span component of the parameter vector that converges to the minimum norm solution and the out-of-span component that saturates. Our experiments on over-parameterized linear regression and deep neural networks support this theory.