Solving Inverse Problems with Deep Linear Neural Networks: Global Convergence Guarantees for Gradient Descent with Weight Decay

TL;DR

This paper proves that deep linear neural networks trained with gradient descent and weight decay can implicitly adapt to unknown low-dimensional structures in inverse problems, ensuring convergence and accurate solutions.

Contribution

It provides the first theoretical guarantees that overparameterized deep linear networks with weight decay adapt to latent structures in inverse problems.

Findings

Deep linear networks converge to approximate solution mappings.

Weight decay regularization aids in adapting to latent subspace structures.

Overparameterization accelerates convergence and improves generalization.

Abstract

Machine learning methods are commonly used to solve inverse problems, wherein an unknown signal must be estimated from few indirect measurements generated via a known acquisition procedure. In particular, neural networks perform well empirically but have limited theoretical guarantees. In this work, we study an underdetermined linear inverse problem that admits several possible solution operators that map measurements to estimates of the target signal. A standard remedy (e.g., in compressed sensing) for establishing the uniqueness of the solution mapping is to assume the existence of a latent low-dimensional structure in the source signal. We ask the following question: do deep linear neural networks adapt to unknown low-dimensional structure when trained by gradient descent with weight decay regularization? We prove that mildly overparameterized deep linear networks trained in this manner converge to an approximate solution mapping that accurately solves the inverse problem while implicitly encoding latent subspace structure. We show rigorously that deep linear networks trained with weight decay automatically adapt to latent subspace structure in the data under practical stepsize and weight initialization schemes. Our work highlights that regularization and overparameterization improve generalization, while overparameterization also accelerates convergence during training.