New insights and perspectives on the natural gradient method

TL;DR

This paper critically analyzes the natural gradient method, revealing its connection to second-order optimization and proposing improvements for practical, robust implementation based on theoretical insights.

Contribution

It offers a new perspective on natural gradient as a second-order method, analyzes convergence, and discusses regularization and invariance properties for better optimization strategies.

Findings

Fisher information matrix often equivalent to Generalized Gauss-Newton matrix

Natural gradient can be viewed as a second-order optimization method

Regularization techniques improve natural gradient robustness

Abstract

Natural gradient descent is an optimization method traditionally motivated from the perspective of information geometry, and works well for many applications as an alternative to stochastic gradient descent. In this paper we critically analyze this method and its properties, and show how it can be viewed as a type of 2nd-order optimization method, with the Fisher information matrix acting as a substitute for the Hessian. In many important cases, the Fisher information matrix is shown to be equivalent to the Generalized Gauss-Newton matrix, which both approximates the Hessian, but also has certain properties that favor its use over the Hessian. This perspective turns out to have significant implications for the design of a practical and robust natural gradient optimizer, as it motivates the use of techniques like trust regions and Tikhonov regularization. Additionally, we make a series of contributions to the understanding of natural gradient and 2nd-order methods, including: a thorough analysis of the convergence speed of stochastic natural gradient descent (and more general stochastic 2nd-order methods) as applied to convex quadratics, a critical examination of the oft-used "empirical" approximation of the Fisher matrix, and an analysis of the (approximate) parameterization invariance property possessed by natural gradient methods (which we show also holds for certain other curvature, but notably not the Hessian).