Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networks

TL;DR

This paper analyzes the local geometric landscape of the regularized loss in deep linear networks, providing error bounds and convergence guarantees for gradient descent.

Contribution

It offers a closed-form characterization of critical points and establishes an error bound that leads to linear convergence analysis for first-order methods.

Findings

Gradient descent converges linearly to critical points.

The error bound relates gradient norm to distance from critical points.

Theoretical results are supported by numerical experiments.

Abstract

The optimization foundations of deep linear networks have recently received significant attention. However, due to their inherent non-convexity and hierarchical structure, analyzing the loss functions of deep linear networks remains a challenging task. In this work, we study the local geometric landscape of the regularized squared loss of deep linear networks around each critical point. Specifically, we derive a closed-form characterization of the critical point set and establish an error bound for the regularized loss under mild conditions on network width and regularization parameters. Notably, this error bound quantifies the distance from a point to the critical point set in terms of the current gradient norm, which can be used to derive linear convergence of first-order methods. To support our theoretical findings, we conduct numerical experiments and demonstrate that gradient descent converges linearly to a critical point when optimizing the regularized loss of deep linear networks.