SuRe: Surprise-Driven Prioritised Replay for Continual LLM Learning

TL;DR

This paper introduces SuRe, a surprise-driven prioritised replay method for continual learning in large language models, combining surprise-based selection with a dual-learner design to improve knowledge retention and adaptation.

Contribution

It proposes a novel surprise-based replay strategy and a dual-learner architecture with EMA merging, significantly enhancing continual learning performance in LLMs.

Findings

SuRe achieves state-of-the-art results in large number of tasks setting.

The dual-learner design improves stability and adaptation speed.

Method remains effective with reduced replay frequency and small buffers.

Abstract

Continual learning, one's ability to adapt to a sequence of tasks without forgetting previously acquired knowledge, remains a major challenge in machine learning and a key gap between artificial and human intelligence. While regularisation and replay perform well in vision, they lag behind multi-task learning for large language models (LLMs), especially at scale with many tasks. We revisit replay and argue that two failure modes drive this gap: selection (what to rehearse) and integration (how to consolidate new knowledge). To address selection, we propose Surprise-prioritised Replay (SuRe), a simple, architecture-agnostic rule that ranks and stores the most surprising (high Negative Log-Likelihood) sequences. SuRe achieves state-of-the-art performance in the Large Number of Tasks (LNT) setting and delivers the best overall average across both Standard CL and LNT benchmarks. To address integration, we add a dual-learner design with fast and slow LoRA adapters merged via an exponential moving average (EMA), enabling rapid adaptation while stabilising long-term knowledge. Combining SuRe with the dual learner yields further gains, including improvements of up to +5 accuracy points on LNT over prior SOTA. Ablation studies confirm that our proposed method remains robust under reduced replay frequency and small buffer size, demonstrating both effectiveness and sample efficiency. Taken together, our results establish replay as a strong baseline for continual LLM fine-tuning and demonstrate that surprise-based selection and slow-weight consolidation are complementary components for mitigating catastrophic forgetting.