Norm-Hierarchy Transitions in Representation Learning: When and Why Neural Networks Abandon Shortcuts

TL;DR

This paper introduces the Norm-Hierarchy Transition framework to explain why neural networks initially rely on shortcuts before transitioning to structured representations, highlighting the role of parameter norm traversal during training.

Contribution

The paper proposes a new theoretical framework that links delayed representation learning to the slow traversal of parameter norm hierarchies during regularized optimization.

Findings

Transition delay grows logarithmically with the ratio of shortcut to structured norms.

Experiments support the framework's predictions across multiple datasets.

Norm hierarchy traversal explains phenomena like grokking and shortcut learning.

Abstract

Neural networks often rely on spurious shortcuts for many epochs before discovering structured representations. However, the mechanism governing when this transition occurs and whether its timing can be predicted remains unclear. Prior work shows that gradient descent converges to low norm solutions and that neural networks exhibit simplicity bias, but neither explains the timescale of the transition from shortcut features to structured representations. We introduce the Norm-Hierarchy Transition (NHT) framework, which explains delayed representation learning as the slow traversal of a hierarchy of parameter norms during regularized optimization. When multiple interpolating solutions exist with different norms, weight decay gradually moves the model from high norm shortcut solutions toward lower norm structured representations. We derive a tight bound showing that the transition delay grows logarithmically with the ratio between shortcut and structured norms. Experiments on modular arithmetic, CIFAR-10 with spurious features, CelebA, and Waterbirds support the predictions of the framework. The results suggest that grokking, shortcut learning, and delayed feature discovery arise from a common mechanism based on norm hierarchy traversal during training.