Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithm

TL;DR

This paper develops a theoretical framework to understand how language models acquire and retain factual knowledge during continual pre-training, and proposes a novel data replay method called STOC to improve knowledge retention.

Contribution

It introduces a unified theoretical analysis of CPT mechanisms, revealing the limitations of regularization and the benefits of data replay, and proposes STOC for better factual knowledge retention.

Findings

Regularization methods do not change the inherent forgetting tendency.

Data replay methods shift convergence dynamics and stabilize knowledge.

STOC effectively mitigates catastrophic forgetting in experiments.

Abstract

Continual Pre-Training (CPT) is essential for enabling Language Models (LMs) to integrate new knowledge without erasing old. While classical CPT techniques like data replay have become the standard paradigm, the mechanisms underlying how LMs acquire and retain facts over time, termed as continual Factual Knowledge Acquisition (cFKA), remain unclear. In this work, we present a theoretical framework that characterizes the training dynamics of cFKA using a single-layer Transformer, offering a unified explanation for the behavior of representative CPT methods. Our analysis reveals that regularization-based methods merely adjust the convergence rate of parameters without altering the inherent forgetting tendency, whereas data replay methods succeed in shifting convergence dynamics and stabilizing pretrained knowledge. Building on these insights, we propose a novel generative data replay approach, called \textbf{S}electing \textbf{T}okens via attenti\textbf{O}n \textbf{C}ontribution~(STOC), which identifies influential factual snippets to guide replay data generation. Extensive experiments on both synthetic and real-world datasets validate our findings and demonstrate that STOC effectively enhances cFKA by mitigating catastrophic forgetting.