A Faster Path to Continual Learning

TL;DR

This paper introduces C-Flat Turbo, an optimized version of C-Flat for continual learning that reduces training overhead by skipping redundant gradient computations, achieving faster training with comparable or better accuracy.

Contribution

It proposes a novel gradient skipping technique and adaptive scheduling to significantly accelerate C-Flat without sacrificing performance in continual learning.

Findings

C-Flat Turbo is 1.0× to 1.25× faster than C-Flat.

It maintains or improves accuracy across various continual learning methods.

The approach reduces gradient computation overhead in the optimization process.

Abstract

Continual Learning (CL) aims to train neural networks on a dynamic stream of tasks without forgetting previously learned knowledge. Among optimization-based approaches, C-Flat has emerged as a promising solution due to its plug-and-play nature and its ability to encourage uniformly low-loss regions for both new and old tasks. However, C-Flat requires three additional gradient computations per iteration, imposing substantial overhead on the optimization process. In this work, we propose C-Flat Turbo, a faster yet stronger optimizer that significantly reduces the training cost. We show that the gradients associated with first-order flatness contain direction-invariant components relative to the proxy-model gradients, enabling us to skip redundant gradient computations in the perturbed ascent steps. Moreover, we observe that these flatness-promoting gradients progressively stabilize across tasks, which motivates a linear scheduling strategy with an adaptive trigger to allocate larger turbo steps for later tasks. Experiments show that C-Flat Turbo is 1.0$\times$ to 1.25$\times$ faster than C-Flat across a wide range of CL methods, while achieving comparable or even improved accuracy.