Implicit regularization in AI meets generalized hardness of approximation in optimization -- Sharp results for diagonal linear networks

TL;DR

This paper investigates the implicit regularization effects of gradient flow in diagonal linear networks, linking it to phase transitions in generalized hardness of approximation, and provides sharp convergence results and characterizations of solutions.

Contribution

It offers the first sharp analysis of implicit regularization in DLNs, connecting it to GHA phenomena and characterizing the selected minimizers based on network depth.

Findings

Gradient flow approximates basis pursuit minimizers.

Convergence bounds depend on initialization size.

Characterization of which minimizer is chosen based on network depth.

Abstract

Understanding the implicit regularization imposed by neural network architectures and gradient based optimization methods is a key challenge in deep learning and AI. In this work we provide sharp results for the implicit regularization imposed by the gradient flow of Diagonal Linear Networks (DLNs) in the over-parameterized regression setting and, potentially surprisingly, link this to the phenomenon of phase transitions in generalized hardness of approximation (GHA). GHA generalizes the phenomenon of hardness of approximation from computer science to, among others, continuous and robust optimization. It is well-known that the $\ell^1$-norm of the gradient flow of DLNs with tiny initialization converges to the objective function of basis pursuit. We improve upon these results by showing that the gradient flow of DLNs with tiny initialization approximates minimizers of the basis pursuit optimization problem (as opposed to just the objective function), and we obtain new and sharp convergence bounds w.r.t.\ the initialization size. Non-sharpness of our results would imply that the GHA phenomenon would not occur for the basis pursuit optimization problem -- which is a contradiction -- thus implying sharpness. Moreover, we characterize $\textit{which}$ $\ell_1$ minimizer of the basis pursuit problem is chosen by the gradient flow whenever the minimizer is not unique. Interestingly, this depends on the depth of the DLN.