Progress Measures for Grokking on Real-world Tasks

TL;DR

This paper investigates the phenomenon of grokking in real-world datasets, introducing new progress measures that better correlate with generalization than traditional weight norm metrics.

Contribution

It introduces three novel progress measures—activation sparsity, weight entropy, and local circuit complexity—that improve understanding of grokking beyond weight norms.

Findings

Grokking can occur outside the expected weight norm ranges.

New progress measures show stronger correlation with grokking than weight norms.

Weight norms are not the primary cause of grokking.

Abstract

Grokking, a phenomenon where machine learning models generalize long after overfitting, has been primarily observed and studied in algorithmic tasks. This paper explores grokking in real-world datasets using deep neural networks for classification under the cross-entropy loss. We challenge the prevalent hypothesis that the $L_2$ norm of weights is the primary cause of grokking by demonstrating that grokking can occur outside the expected range of weight norms. To better understand grokking, we introduce three new progress measures: activation sparsity, absolute weight entropy, and approximate local circuit complexity. These measures are conceptually related to generalization and demonstrate a stronger correlation with grokking in real-world datasets compared to weight norms. Our findings suggest that while weight norms might usually correlate with grokking and our progress measures, they are not causative, and our proposed measures provide a better understanding of the dynamics of grokking.