On the ISS Property of the Gradient Flow for Single Hidden-Layer Neural Networks with Linear Activations

TL;DR

This paper analyzes how overparameterization affects the robustness of gradient flow in single hidden-layer linear neural networks, revealing conditions for stability and the presence of spurious equilibria.

Contribution

It provides the first analysis of the ISS property for gradient flow in overparameterized linear neural networks with one-dimensional input and output.

Findings

Derived sufficient conditions for robustness based on convergence criteria.

Identified spurious equilibria outside the loss minimization set.

Discussed potential extensions to more general neural network architectures.

Abstract

Recent research in neural networks and machine learning suggests that using many more parameters than strictly required by the initial complexity of a regression problem can result in more accurate or faster-converging models -- contrary to classical statistical belief. This phenomenon, sometimes known as ``benign overfitting'', raises questions regarding in what other ways might overparameterization affect the properties of a learning problem. In this work, we investigate the effects of overfitting on the robustness of gradient-descent training when subject to uncertainty on the gradient estimation. This uncertainty arises naturally if the gradient is estimated from noisy data or directly measured. Our object of study is a linear neural network with a single, arbitrarily wide, hidden layer and an arbitrary number of inputs and outputs. In this paper we solve the problem for the case where the input and output of our neural-network are one-dimensional, deriving sufficient conditions for robustness of our system based on necessary and sufficient conditions for convergence in the undisturbed case. We then show that the general overparametrized formulation introduces a set of spurious equilibria which lay outside the set where the loss function is minimized, and discuss directions of future work that might extend our current results for more general formulations.