How to explain grokking

TL;DR

This paper models the phenomenon of grokking, or delayed generalization in learning, using stochastic gradient Langevin dynamics and thermodynamic principles to provide an explanation.

Contribution

It introduces a novel thermodynamic framework to understand grokking through stochastic gradient Langevin dynamics modeling.

Findings

Grokking can be explained by thermodynamic principles.

Stochastic gradient Langevin dynamics captures delayed generalization.

The approach offers new insights into learning dynamics.

Abstract

Explanation of grokking (delayed generalization) in learning is given by modeling grokking by the stochastic gradient Langevin dynamics (Brownian motion) and applying the ideas of thermodynamics.