Topological Signatures of Grokking

TL;DR

This paper uses persistent homology to identify topological signatures associated with the grokking phenomenon in neural networks, revealing structural changes linked to generalization.

Contribution

It introduces a topological analysis framework using persistent homology to characterize neural network representations during grokking, highlighting a clear topological signature.

Findings

A sharp increase in persistence of first homology ($H_1$) correlates with grokking.

Emergence of a dominant long-lived topological feature reflects cyclic structure.

Topological transitions are tied to generalization, not memorization.

Abstract

We study the grokking phenomenon through the lens of topology. Using persistent homology on point clouds derived from the embedding matrices of a range of models trained on modular arithmetic with varying primes, we identify a clear and consistent topological signature of grokking: a sharp increase in both the maximum and total persistence of first homology ($H_1$). Persistence diagrams reveal the emergence of a dominant long-lived topological feature together with increasingly structured secondary features, reflecting the underlying cyclic structure of the task. Compared to existing spectral and geometric diagnostics -- specifically, Fourier analysis and local intrinsic dimension -- persistent homology provides a unified geometric and topological characterization of representation learning, capturing both local and global multi-scale structure. Ablations across data regimes and control settings show that these topological transitions are tied to generalization rather than memorization. Our results suggest that persistent homology offers a principled and interpretable framework for analyzing how neural networks internalize latent structure during training.