Online Curvature-Aware Replay: Leveraging $\mathbf{2^{nd}}$ Order Information for Online Continual Learning

TL;DR

This paper introduces OCAR, a second-order optimization method for online continual learning that uses Fisher Information Matrix approximations to stabilize learning and prevent forgetting in nonstationary data streams.

Contribution

The paper formalizes replay-based online continual learning as a second-order joint optimization and proposes OCAR, leveraging FIM for improved stability and performance.

Findings

OCAR outperforms state-of-the-art methods in continual learning benchmarks.

FIM-based preconditioning stabilizes learning in non-iid data streams.

Tikhonov regularization plays a key role in balancing stability and plasticity.

Abstract

Online Continual Learning (OCL) models continuously adapt to nonstationary data streams, usually without task information. These settings are complex and many traditional CL methods fail, while online methods (mainly replay-based) suffer from instabilities after the task shift. To address this issue, we formalize replay-based OCL as a second-order online joint optimization with explicit KL-divergence constraints on replay data. We propose Online Curvature-Aware Replay (OCAR) to solve the problem: a method that leverages second-order information of the loss using a K-FAC approximation of the Fisher Information Matrix (FIM) to precondition the gradient. The FIM acts as a stabilizer to prevent forgetting while also accelerating the optimization in non-interfering directions. We show how to adapt the estimation of the FIM to a continual setting stabilizing second-order optimization for non-iid data, uncovering the role of the Tikhonov regularization in the stability-plasticity tradeoff. Empirical results show that OCAR outperforms state-of-the-art methods in continual metrics achieving higher average accuracy throughout the training process in three different benchmarks.