Black Box Lie Group Preconditioners for SGD

TL;DR

This paper introduces Lie group-based preconditioners for stochastic gradient descent that leverage curvature information, are matrix-free, and maintain invariance properties, leading to more robust and efficient optimization.

Contribution

It proposes novel preconditioners constrained on Lie groups, enabling efficient curvature exploitation without line search or damping, improving SGD convergence.

Findings

Preconditioners accelerate SGD convergence.

Lie group constraints preserve symmetry and invariance.

Default hyperparameters perform well across tasks.

Abstract

A matrix free and a low rank approximation preconditioner are proposed to accelerate the convergence of stochastic gradient descent (SGD) by exploiting curvature information sampled from Hessian-vector products or finite differences of parameters and gradients similar to the BFGS algorithm. Both preconditioners are fitted with an online updating manner minimizing a criterion that is free of line search and robust to stochastic gradient noise, and further constrained to be on certain connected Lie groups to preserve their corresponding symmetry or invariance, e.g., orientation of coordinates by the connected general linear group with positive determinants. The Lie group's equivariance property facilitates preconditioner fitting, and its invariance property saves any need of damping, which is common in second-order optimizers, but difficult to tune. The learning rate for parameter updating and step size for preconditioner fitting are naturally normalized, and their default values work well in most situations.