To Each Optimizer a Norm, To Each Norm its Generalization

TL;DR

This paper investigates the implicit regularization effects of optimization methods in linear models, revealing how different norms influence the solutions and their generalization, and proposes techniques to bias optimizers towards better solutions.

Contribution

It introduces a framework linking optimization methods to the norms they implicitly minimize, providing new insights into generalization in linear models and extending to non-linear models via neural tangent kernels.

Findings

Different interpolating solutions correspond to different norms.

Projections can move between solutions in over-parameterized models.

Biasing optimizers towards certain norms improves test performance.

Abstract

We study the implicit regularization of optimization methods for linear models interpolating the training data in the under-parametrized and over-parametrized regimes. Since it is difficult to determine whether an optimizer converges to solutions that minimize a known norm, we flip the problem and investigate what is the corresponding norm minimized by an interpolating solution. Using this reasoning, we prove that for over-parameterized linear regression, projections onto linear spans can be used to move between different interpolating solutions. For under-parameterized linear classification, we prove that for any linear classifier separating the data, there exists a family of quadratic norms ||.||_P such that the classifier's direction is the same as that of the maximum P-margin solution. For linear classification, we argue that analyzing convergence to the standard maximum l2-margin is arbitrary and show that minimizing the norm induced by the data results in better generalization. Furthermore, for over-parameterized linear classification, projections onto the data-span enable us to use techniques from the under-parameterized setting. On the empirical side, we propose techniques to bias optimizers towards better generalizing solutions, improving their test performance. We validate our theoretical results via synthetic experiments, and use the neural tangent kernel to handle non-linear models.