ONG: Orthogonal Natural Gradient Descent

TL;DR

This paper introduces ONG, a novel continual learning method that integrates natural gradients with orthogonal projections to improve convergence and performance across tasks.

Contribution

It proposes the Orthogonal Natural Gradient Descent (ONG) algorithm, combining natural gradient preconditioning with orthogonal projections for continual learning.

Findings

Preliminary results on MNIST benchmarks show potential of ONG.

Naive combination of natural gradients and orthogonal projections has issues.

Future work aims to improve theoretical foundations and empirical validation.

Abstract

Orthogonal Gradient Descent (OGD) has emerged as a powerful method for continual learning. However, its Euclidean projections do not leverage the underlying information-geometric structure of the problem, which can lead to suboptimal convergence in learning tasks. To address this, we propose incorporating the natural gradient into OGD and present \textbf{ONG (Orthogonal Natural Gradient Descent)}. ONG preconditions each new task-specific gradient with an efficient EKFAC approximation of the inverse Fisher information matrix, yielding updates that follow the steepest descent direction under a Riemannian metric. To preserve performance on previously learned tasks, ONG projects these natural gradients onto the orthogonal complement of prior tasks' natural gradients. We provide an initial theoretical justification for this procedure, introduce the Orthogonal Natural Gradient Descent (ONG) algorithm, and present preliminary results on the Permuted and Rotated MNIST benchmarks. Our preliminary results, however, indicate that a naive combination of natural gradients and orthogonal projections has potential issues. This finding has motivated continued future work focused on robustly reconciling these geometric perspectives to develop a continual learning method, establishing a more rigorous theoretical foundation with formal convergence guarantees, and extending empirical validation to large-scale continual learning benchmarks.