On the Impact of Stable Ranks in Deep Nets

TL;DR

This paper explores how stable ranks of layer weights influence the behavior, training, and expressivity of deep neural networks, revealing layerwise effects and potential benefits for convergence speed.

Contribution

It introduces a new random DNN model based on stable rank sampling and analyzes the impact of stable ranks on network dynamics and training.

Findings

Stable ranks act as linear factors affecting layerwise behavior.

Training dynamics are influenced by stable rank constraints, especially in overparameterized regimes.

Stable rank initialization can accelerate convergence.

Abstract

A recent line of work has established intriguing connections between the generalization/compression properties of a deep neural network (DNN) model and the so-called layer weights' stable ranks. Intuitively, the latter are indicators of the effective number of parameters in the net. In this work, we address some natural questions regarding the space of DNNs conditioned on the layers' stable rank, where we study feed-forward dynamics, initialization, training and expressivity. To this end, we first propose a random DNN model with a new sampling scheme based on stable rank. Then, we show how feed-forward maps are affected by the constraint and how training evolves in the overparametrized regime (via Neural Tangent Kernels). Our results imply that stable ranks appear layerwise essentially as linear factors whose effect accumulates exponentially depthwise. Moreover, we provide empirical analysis suggesting that stable rank initialization alone can lead to convergence speed ups.