Get rid of your constraints and reparametrize: A study in NNLS and implicit bias

TL;DR

This paper explores reparametrization and Riemannian optimization techniques to solve non-negative least squares efficiently, demonstrating global convergence, accelerated methods, and stability, with implications for understanding implicit bias in neural networks.

Contribution

It introduces a novel reparametrization approach connecting gradient descent to Riemannian optimization for NNLS, achieving global convergence and accelerated methods without geodesic calculations.

Findings

Global convergence of gradient flow on reparametrized objectives

Accelerated convergence using second-order ODEs

Stability against negative perturbations

Abstract

Over the past years, there has been significant interest in understanding the implicit bias of gradient descent optimization and its connection to the generalization properties of overparametrized neural networks. Several works observed that when training linear diagonal networks on the square loss for regression tasks (which corresponds to overparametrized linear regression) gradient descent converges to special solutions, e.g., non-negative ones. We connect this observation to Riemannian optimization and view overparametrized GD with identical initialization as a Riemannian GD. We use this fact for solving non-negative least squares (NNLS), an important problem behind many techniques, e.g., non-negative matrix factorization. We show that gradient flow on the reparametrized objective converges globally to NNLS solutions, providing convergence rates also for its discretized counterpart. Unlike previous methods, we do not rely on the calculation of exponential maps or geodesics. We further show accelerated convergence using a second-order ODE, lending itself to accelerated descent methods. Finally, we establish the stability against negative perturbations and discuss generalization to other constrained optimization problems.