Spring-block theory of feature learning in deep neural networks

TL;DR

This paper introduces a phase diagram and a mechanical theory to explain how deep neural networks learn features, linking layer dynamics to generalization performance.

Contribution

It presents a novel noise-nonlinearity phase diagram and a macroscopic mechanical theory that elucidates feature learning across layers in deep networks.

Findings

Identifies regimes where shallow or deep layers learn more effectively.

Links feature learning to generalization through a mechanical theory.

Reproduces the phase diagram with the proposed theory.

Abstract

Feature-learning deep nets progressively collapse data to a regular low-dimensional geometry. How this emerges from the collective action of nonlinearity, noise, learning rate, and other factors, has eluded first-principles theories built from microscopic neuronal dynamics. We exhibit a noise-nonlinearity phase diagram that identifies regimes where shallow or deep layers learn more effectively and propose a macroscopic mechanical theory that reproduces the diagram and links feature learning across layers to generalization.