Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models

TL;DR

This paper investigates continual pre-training of large language models, demonstrating that experience replay and gradient alignment improve stability and prevent forgetting across multiple languages and scales, with efficient implementations and insights on resource allocation.

Contribution

It introduces the first application of gradient alignment techniques in LLM pre-training and proposes an efficient meta-experience replay method to enhance continual learning.

Findings

Experience replay and gradient alignment improve stability without forgetting.

Small replay rates are more resource-efficient than increasing model size.

Gradient alignment techniques are effective in large-scale LLM pre-training.

Abstract

Training large language models (LLMs) typically involves pre-training on massive corpora, only to restart the process entirely when new data becomes available. A more efficient and resource-conserving approach would be continual pre-training, where models are updated with new data rather than retraining from scratch. However, the introduction of new data often causes distribution shifts, leading to performance degradation on previously learned tasks. In this paper, we take a deeper look at two popular proposals for addressing this distribution shift within the continual learning literature: experience replay and gradient alignment. We consider continual pre-training of models within the Llama family of architectures at a large scale across languages with 100 billion tokens of training data in each language, finding that both replay and gradient alignment lead to more stable learning without forgetting. This conclusion holds both as we vary the model scale and as we vary the number and diversity of tasks. Moreover, we are the first to demonstrate the effectiveness of gradient alignment techniques in the context of LLM pre-training and propose an efficient implementation of meta-experience replay (MER) that imbues experience replay with the benefits of gradient alignment despite negligible compute and memory overhead. Our scaling analysis across model sizes and replay rates indicates that small rates of replaying old examples are definitely a more valuable use of compute than investing in model size, but that it is more compute efficient to scale the size of the model than invest in high rates of replaying old examples.