Sufficient Conditions for Stability of Minimum-Norm Interpolating Deep ReLU Networks

TL;DR

This paper analyzes the stability of minimum-norm interpolating deep ReLU networks, identifying conditions under which such networks are stable, especially emphasizing the role of low-rank layers and stable sub-networks.

Contribution

It provides new sufficient conditions for the stability of overparameterized deep ReLU networks trained to zero training error, highlighting the importance of low-rank structures.

Findings

Stable sub-networks contribute to overall stability.

Low-rank layers are crucial for stability in deep networks.

Networks without low-rank layers may lack stability.

Abstract

Algorithmic stability is a classical framework for analyzing the generalization error of learning algorithms. It predicts that an algorithm has small generalization error if it is insensitive to small perturbations in the training set such as the removal or replacement of a training point. While stability has been demonstrated for numerous well-known algorithms, this framework has had limited success in analyses of deep neural networks. In this paper we study the algorithmic stability of deep ReLU homogeneous neural networks that achieve zero training error using parameters with the smallest $L_2$ norm, also known as the minimum-norm interpolation, a phenomenon that can be observed in overparameterized models trained by gradient-based algorithms. We investigate sufficient conditions for such networks to be stable. We find that 1) such networks are stable when they contain a (possibly small) stable sub-network, followed by a layer with a low-rank weight matrix, and 2) such networks are not guaranteed to be stable even when they contain a stable sub-network, if the following layer is not low-rank. The low-rank assumption is inspired by recent empirical and theoretical results which demonstrate that training deep neural networks is biased towards low-rank weight matrices, for minimum-norm interpolation and weight-decay regularization.