SCALE: Upscaled Continual Learning of Large Language Models

TL;DR

SCALE introduces a width upscaling architecture for large language models that preserves pre-trained knowledge while efficiently acquiring new information through selective expansion and training strategies.

Contribution

The paper proposes a novel width upscaling method, SCALE, that maintains model stability during continual learning by combining preservation and adaptation principles with lightweight expansions.

Findings

SCALE reduces forgetting in synthetic and real-world benchmarks.

SCALE achieves competitive performance on Korean language tasks.

The approach stabilizes optimization compared to standard continual learning methods.

Abstract

We revisit continual pre-training for large language models and argue that progress now depends more on scaling the right structure than on scaling parameters alone. We introduce SCALE, a width upscaling architecture that inserts lightweight expansion into linear modules while freezing all pre-trained parameters. This preserves the residual and attention topologies and increases capacity without perturbing the base model's original functionality. SCALE is guided by two principles: Persistent Preservation, which maintains the base model's behavior via preservation-oriented initialization and freezing of the pre-trained weights, and Collaborative Adaptation, which selectively trains a subset of expansion components to acquire new knowledge with minimal interference. We instantiate these ideas as SCALE-Preserve (preservation-first), SCALE-Adapt (adaptation-first), and SCALE-Route, an optional routing extension that performs token-level routing between preservation and adaptation heads. On a controlled synthetic biography benchmark, SCALE mitigates the severe forgetting observed with depth expansion while still acquiring new knowledge. In continual pre-training on a Korean corpus, SCALE variants achieve less forgetting on English evaluations and competitive gains on Korean benchmarks, with these variants offering the best overall stability-plasticity trade-off. Accompanying analysis clarifies when preservation provably holds and why the interplay between preservation and adaptation stabilizes optimization compared to standard continual learning setups.