Unified View of Grokking, Double Descent and Emergent Abilities: A Perspective from Circuits Competition

TL;DR

This paper presents a unified framework explaining grokking, double descent, and emergent abilities in neural networks through the competition between memorization and generalization circuits, extending to various model sizes and training data.

Contribution

It introduces a comprehensive framework that unifies understanding of multiple phenomena in deep learning, including new predictions and extensions to multi-task learning.

Findings

Detailed analysis of double descent phenomenon

Two verifiable predictions about double descent occurrence

Extension of framework to multi-task learning and emergent abilities

Abstract

Recent studies have uncovered intriguing phenomena in deep learning, such as grokking, double descent, and emergent abilities in large language models, which challenge human intuition and are crucial for a deeper understanding of neural models. In this paper, we present a comprehensive framework that provides a unified view of these three phenomena, focusing on the competition between memorization and generalization circuits. This approach, initially employed to explain grokking, is extended in our work to encompass a wider range of model sizes and training data volumes. Our framework delineates four distinct training dynamics, each depending on varying combinations of model size and training data quantity. Utilizing this framework, we provide a detailed analysis of the double descent phenomenon and propose two verifiable predictions regarding its occurrence, both substantiated by our experimental results. Moreover, we expand our framework to the multi-task learning paradigm, demonstrating how algorithm tasks can be turned into emergent abilities. This offers a novel perspective to understand emergent abilities in Large Language Models.