C-Flat++: Towards a More Efficient and Powerful Framework for Continual Learning

TL;DR

C-Flat++ introduces a flexible framework for continual learning that promotes flatter loss landscapes to enhance stability and performance, with an efficient variant reducing computational costs.

Contribution

The paper proposes C-Flat, a novel method promoting flatter minima in continual learning, and C-Flat++, an efficient extension reducing update costs while maintaining effectiveness.

Findings

C-Flat improves performance across various continual learning settings.

C-Flat++ significantly reduces update costs compared to C-Flat.

The methods are compatible with multiple CL paradigms and datasets.

Abstract

Balancing sensitivity to new tasks and stability for retaining past knowledge is crucial in continual learning (CL). Recently, sharpness-aware minimization has proven effective in transfer learning and has also been adopted in continual learning (CL) to improve memory retention and learning efficiency. However, relying on zeroth-order sharpness alone may favor sharper minima over flatter ones in certain settings, leading to less robust and potentially suboptimal solutions. In this paper, we propose \textbf{C}ontinual \textbf{Flat}ness (\textbf{C-Flat}), a method that promotes flatter loss landscapes tailored for CL. C-Flat offers plug-and-play compatibility, enabling easy integration with minimal modifications to the code pipeline. Besides, we present a general framework that integrates C-Flat into all major CL paradigms and conduct comprehensive comparisons with loss-minima optimizers and flat-minima-based CL methods. Our results show that C-Flat consistently improves performance across a wide range of settings. In addition, we introduce C-Flat++, an efficient yet effective framework that leverages selective flatness-driven promotion, significantly reducing the update cost required by C-Flat. Extensive experiments across multiple CL methods, datasets, and scenarios demonstrate the effectiveness and efficiency of our proposed approaches. Code is available at https://github.com/WanNaa/C-Flat.