Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning

TL;DR

This paper investigates the training dynamics of language models, revealing loss deceleration caused by zero-sum learning, and shows how scaling mitigates this effect, offering insights to improve model training.

Contribution

It introduces the concept of zero-sum learning as a cause of loss deceleration and demonstrates how scaling alleviates this issue in language model training.

Findings

Loss deceleration occurs early in training with piecewise linear loss curves.

Scaling reduces the loss at which deceleration occurs and improves post-deceleration loss improvement rates.

Zero-sum learning causes destructive interference among per-example gradients, hindering training progress.

Abstract

This work aims to understand how scaling improves language models, specifically in terms of training dynamics. We find that language models undergo loss deceleration early in training; an abrupt slowdown in the rate of loss improvement, resulting in piecewise linear behaviour of the loss curve in log-log space. Scaling up the model mitigates this transition by (1) decreasing the loss at which deceleration occurs, and (2) improving the log-log rate of loss improvement after deceleration. We attribute loss deceleration to a type of degenerate training dynamics we term zero-sum learning (ZSL). In ZSL, per-example gradients become systematically opposed, leading to destructive interference in per-example changes in loss. As a result, improving loss on one subset of examples degrades it on another, bottlenecking overall progress. Loss deceleration and ZSL provide new insights into the training dynamics underlying language model scaling laws, and could potentially be targeted directly to improve language models independent of scale. We make our code and artefacts available at: https://github.com/mirandrom/zsl