Scalable Strategies for Continual Learning with Replay

TL;DR

This paper introduces scalable replay strategies for continual learning, combining low rank adaptation, consolidation, and sequential merging to reduce replay samples and improve model performance across many tasks.

Contribution

It presents a unified, scalable toolkit for continual learning that integrates replay with model merging and low rank adaptation techniques.

Findings

Consolidation reduces replay samples by up to 55%.

Sequential merging effectively combines tasks in continual learning.

Combined strategies outperform individual methods in scalability and performance.

Abstract

Future deep learning models will be distinguished by systems that perpetually learn through interaction, imagination, and cooperation, blurring the line between training and inference. This makes continual learning a critical challenge, as methods that efficiently maximize bidirectional transfer across learning trajectories will be essential. Replay is on track to play a foundational role in continual learning, allowing models to directly reconcile new information with past knowledge. In practice, however, replay is quite unscalable, doubling the cost of continual learning when applied naively. Moreover, the continual learning literature has not fully synchronized with the multi-task fine-tuning literature, having not fully integrated highly scalable techniques like model merging and low rank adaptation into a replay-enabled toolset that can produce a unified model in the face of many sequential tasks. In this paper, we begin by applying and analyzing low rank adaptation in a continual learning setting. Next, we introduce consolidation, a phasic approach to replay which leads to up to 55\% less replay samples being needed for a given performance target. Then, we propose sequential merging, an offshoot of task arithmetic which is tailored to the continual learning setting and is shown to work well in combination with replay. Finally, we demonstrate that the developed strategies can operate synergistically, resulting in a highly scalable toolset that outperforms standalone variants.