Experiments with Rich Regime Training for Deep Learning

TL;DR

This paper empirically investigates the rich regime in deep learning, revealing the importance of active parameters in the bottom layers and proposing efficient layer-wise sparse training methods that maintain performance while reducing computation.

Contribution

It introduces static and probabilistic layer-wise sparse SGD methods, demonstrating their effectiveness and efficiency in training deep neural networks in the rich regime.

Findings

Active parameters are concentrated in bottom layers.

Re-initializing active parameters worsens generalization.

Probabilistic LWS-SGD matches vanilla SGD performance.

Abstract

In spite of advances in understanding lazy training, recent work attributes the practical success of deep learning to the rich regime with complex inductive bias. In this paper, we study rich regime training empirically with benchmark datasets, and find that while most parameters are lazy, there is always a small number of active parameters which change quite a bit during training. We show that re-initializing (resetting to their initial random values) the active parameters leads to worse generalization. Further, we show that most of the active parameters are in the bottom layers, close to the input, especially as the networks become wider. Based on such observations, we study static Layer-Wise Sparse (LWS) SGD, which only updates some subsets of layers. We find that only updating the top and bottom layers have good generalization and, as expected, only updating the top layers yields a fast algorithm. Inspired by this, we investigate probabilistic LWS-SGD, which mostly updates the top layers and occasionally updates the full network. We show that probabilistic LWS-SGD matches the generalization performance of vanilla SGD and the back-propagation time can be 2-5 times more efficient.