NG+ : A Multi-Step Matrix-Product Natural Gradient Method for Deep Learning

TL;DR

NG+ introduces a novel multi-step matrix-product natural gradient method that efficiently approximates second-order information for deep learning, demonstrating competitive performance across various tasks.

Contribution

The paper proposes NG+, a new second-order optimization method using a generalized Fisher information matrix in matrix form, with controlled computational cost and theoretical convergence guarantees.

Findings

NG+ achieves competitive results on image classification, quantum chemistry, translation, and recommendation tasks.

The method maintains a fixed GFIM over multiple steps, reducing computational overhead.

Global convergence and regret bounds are established under mild conditions.

Abstract

In this paper, a novel second-order method called NG+ is proposed. By following the rule ``the shape of the gradient equals the shape of the parameter", we define a generalized fisher information matrix (GFIM) using the products of gradients in the matrix form rather than the traditional vectorization. Then, our generalized natural gradient direction is simply the inverse of the GFIM multiplies the gradient in the matrix form. Moreover, the GFIM and its inverse keeps the same for multiple steps so that the computational cost can be controlled and is comparable with the first-order methods. A global convergence is established under some mild conditions and a regret bound is also given for the online learning setting. Numerical results on image classification with ResNet50, quantum chemistry modeling with Schnet, neural machine translation with Transformer and recommendation system with DLRM illustrate that GN+ is competitive with the state-of-the-art methods.