Preconditioner on Matrix Lie Group for SGD

TL;DR

This paper introduces a unified framework for preconditioners in stochastic gradient descent on matrix Lie groups, encompassing many existing methods like Adam and RMSProp, and demonstrates their effectiveness on large-scale machine learning tasks.

Contribution

It proposes a novel unified framework for preconditioners in SGD on matrix Lie groups, connecting existing methods and enabling efficient estimation.

Findings

Preconditioners derived from the framework improve convergence.

Many existing optimizers are special cases of the proposed methods.

Experimental results show enhanced performance on large-scale problems.

Abstract

We study two types of preconditioners and preconditioned stochastic gradient descent (SGD) methods in a unified framework. We call the first one the Newton type due to its close relationship to the Newton method, and the second one the Fisher type as its preconditioner is closely related to the inverse of Fisher information matrix. Both preconditioners can be derived from one framework, and efficiently estimated on any matrix Lie groups designated by the user using natural or relative gradient descent minimizing certain preconditioner estimation criteria. Many existing preconditioners and methods, e.g., RMSProp, Adam, KFAC, equilibrated SGD, batch normalization, etc., are special cases of or closely related to either the Newton type or the Fisher type ones. Experimental results on relatively large scale machine learning problems are reported for performance study.