Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learning

TL;DR

This paper introduces FOPNG, a novel optimizer for continual learning that uses Fisher-orthogonal constraints to prevent forgetting old tasks while learning new ones, unifying natural gradient and orthogonal gradient methods.

Contribution

The paper proposes FOPNG, a new optimizer that enforces Fisher-orthogonal constraints on parameter updates, combining natural gradient descent with orthogonal gradient methods for continual learning.

Findings

FOPNG outperforms existing methods on standard benchmarks.

The approach effectively prevents catastrophic forgetting.

Efficient implementation using diagonal Fisher is demonstrated.

Abstract

Continual learning aims to enable neural networks to acquire new knowledge on sequential tasks. However, the key challenge in such settings is to learn new tasks without catastrophically forgetting previously learned tasks. We propose the Fisher-Orthogonal Projected Natural Gradient Descent (FOPNG) optimizer, which enforces Fisher-orthogonal constraints on parameter updates to preserve old task performance while learning new tasks. Unlike existing methods that operate in Euclidean parameter space, FOPNG projects gradients onto the Fisher-orthogonal complement of previous task gradients. This approach unifies natural gradient descent with orthogonal gradient methods within an information-geometric framework. We provide theoretical analysis deriving the projected update, describe efficient and practical implementations using the diagonal Fisher, and demonstrate strong results on standard continual learning benchmarks such as Permuted-MNIST, Split-MNIST, Rotated-MNIST, Split-CIFAR10, and Split-CIFAR100. Our code is available at https://github.com/ishirgarg/FOPNG.