MSSR: Memory-Aware Adaptive Replay for Continual LLM Fine-Tuning

TL;DR

This paper introduces MSSR, a novel replay framework that adaptively schedules rehearsal based on memory strength to improve continual fine-tuning of large language models, reducing forgetting and enhancing performance.

Contribution

MSSR is a new experience replay method that estimates sample memory strength and adaptively schedules rehearsal, outperforming existing replay strategies in continual LLM fine-tuning.

Findings

MSSR consistently outperforms state-of-the-art replay baselines.

MSSR achieves strong gains on reasoning-intensive and multiple-choice benchmarks.

MSSR effectively mitigates catastrophic forgetting while maintaining fast adaptation.

Abstract

Continual fine-tuning of large language models (LLMs) is becoming increasingly crucial as these models are deployed in dynamic environments where tasks and data distributions evolve over time. While strong adaptability enables rapid acquisition of new knowledge, it also exposes LLMs to catastrophic forgetting, where previously learned skills degrade during sequential training. Existing replay-based strategies, such as fixed interleaved replay, accuracy-supervised, and loss-driven scheduling, remain limited: some depend on heuristic rules and provide only partial mitigation of forgetting, while others improve performance but incur substantial computational overhead. Motivated by retention dynamics under sequential fine-tuning, we propose Memory-Inspired Sampler and Scheduler Replay (MSSR), an experience replay framework that estimates sample-level memory strength and schedules rehearsal at adaptive intervals to mitigate catastrophic forgetting while maintaining fast adaptation. Extensive experiments across three backbone models and 11 sequential tasks show that MSSR consistently outperforms state-of-the-art replay baselines, with particularly strong gains on reasoning-intensive and multiple-choice benchmarks.