Hidden Breakthroughs in Language Model Training

TL;DR

This paper introduces POLCA, a method to uncover hidden conceptual breakthroughs during language model training by decomposing loss changes, revealing interpretable phase transitions that enhance understanding of learning dynamics.

Contribution

The paper presents POLCA, a novel technique for decomposing loss variations to identify hidden, interpretable breakthroughs in model training that are obscured by traditional loss metrics.

Findings

POLCA successfully recovers clusters representing interpretative breakthroughs.

Hidden phase transitions are frequent and can be identified with the proposed method.

The method enhances unsupervised interpretability of training dynamics.

Abstract

Loss curves are smooth during most of model training, so visible discontinuities stand out as possible conceptual breakthroughs. Studying these breakthroughs enables a deeper understanding of learning dynamics, but only when they are properly identified. This paper argues that similar breakthroughs occur frequently throughout training but they are obscured by a loss metric that collapses all variation into a single scalar. To find these hidden transitions, we introduce POLCA, a method for decomposing changes in loss along arbitrary bases of the low-rank training subspace. We use our method to identify clusters of samples that share similar changes in loss during training, disaggregating the overall loss into that of smaller groups of conceptually similar data. We validate our method on synthetic arithmetic and natural language tasks, showing that POLCA recovers clusters that represent interpretable breakthroughs in the model's capabilities. We demonstrate the promise of these hidden phase transitions as a tool for unsupervised interpretability.