Information-Theoretic Progress Measures reveal Grokking is an Emergent Phase Transition

TL;DR

This paper investigates the grokking phenomenon in neural networks as an emergent phase transition, using information theory to analyze collective neuron behavior and identify precursors to generalization.

Contribution

It introduces an information-theoretic framework to characterize grokking as a phase transition and highlights how training techniques influence this emergent behavior.

Findings

Grokking corresponds to an emergent phase transition in neural networks.

Higher-order mutual information reveals distinct training phases before grokking.

Weight decay and initialization can accelerate the onset of grokking.

Abstract

This paper studies emergent phenomena in neural networks by focusing on grokking where models suddenly generalize after delayed memorization. To understand this phase transition, we utilize higher-order mutual information to analyze the collective behavior (synergy) and shared properties (redundancy) between neurons during training. We identify distinct phases before grokking allowing us to anticipate when it occurs. We attribute grokking to an emergent phase transition caused by the synergistic interactions between neurons as a whole. We show that weight decay and weight initialization can enhance the emergent phase.