Is All Learning (Natural) Gradient Descent?

TL;DR

This paper demonstrates that many effective learning rules can be reformulated as natural gradient descent with appropriate metrics, unifying various learning algorithms under a common geometric framework.

Contribution

It shows that a broad class of learning rules can be expressed as natural gradient descent with a canonical and optimal metric, providing a unified geometric perspective.

Findings

Effective learning rules can be rewritten as natural gradient descent.

Identifies canonical and optimal metrics for these learning rules.

Applicable to continuous, discrete, stochastic, and higher-order rules.

Abstract

This paper shows that a wide class of effective learning rules -- those that improve a scalar performance measure over a given time window -- can be rewritten as natural gradient descent with respect to a suitably defined loss function and metric. Specifically, we show that parameter updates within this class of learning rules can be expressed as the product of a symmetric positive definite matrix (i.e., a metric) and the negative gradient of a loss function. We also demonstrate that these metrics have a canonical form and identify several optimal ones, including the metric that achieves the minimum possible condition number. The proofs of the main results are straightforward, relying only on elementary linear algebra and calculus, and are applicable to continuous-time, discrete-time, stochastic, and higher-order learning rules, as well as loss functions that explicitly depend on time.