Superposition unifies power-law training dynamics

TL;DR

This paper demonstrates that feature superposition in neural networks induces a universal power-law training dynamic with an exponent of approximately 1, significantly accelerating training and reducing dependence on data specifics.

Contribution

The study provides an analytic theory showing superposition causes a universal power-law exponent and accelerates training, unifying diverse training behaviors.

Findings

Superposition induces a universal power-law exponent of ~1.

Superposition accelerates training up to tenfold.

Training dynamics become data-independent with superposition.

Abstract

We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training without superposition, establishing that the power-law training exponent depends on both the input data statistics and channel importance. Remarkably, we discover that a superposition bottleneck induces a transition to a universal power-law exponent of $\sim 1$, independent of data and channel statistics. This one over time training with superposition represents an up to tenfold acceleration compared to the purely sequential learning that takes place in the absence of superposition. Our finding that superposition leads to rapid training with a data-independent power law exponent may have important implications for a wide range of neural networks that employ superposition, including production-scale large language models.