How regularization affects the critical points in linear networks

TL;DR

This paper investigates how regularization influences the critical points in linear neural networks, revealing that even minimal regularization can significantly alter the landscape of these points.

Contribution

It introduces an optimal control framework and derives a learning algorithm to characterize and analyze the critical points under regularization in linear networks.

Findings

Critical points are characterized by a nonlinear matrix equation.

Regularization can fundamentally change the critical point structure.

Small regularization levels can lead to significant differences in critical point diagrams.

Abstract

This paper is concerned with the problem of representing and learning a linear transformation using a linear neural network. In recent years, there has been a growing interest in the study of such networks in part due to the successes of deep learning. The main question of this body of research and also of this paper pertains to the existence and optimality properties of the critical points of the mean-squared loss function. The primary concern here is the robustness of the critical points with regularization of the loss function. An optimal control model is introduced for this purpose and a learning algorithm (regularized form of backprop) derived for the same using the Hamilton's formulation of optimal control. The formulation is used to provide a complete characterization of the critical points in terms of the solutions of a nonlinear matrix-valued equation, referred to as the characteristic equation. Analytical and numerical tools from bifurcation theory are used to compute the critical points via the solutions of the characteristic equation. The main conclusion is that the critical point diagram can be fundamentally different even with arbitrary small amounts of regularization.