Quantitative Convergences of Lie Group Momentum Optimizers

TL;DR

This paper introduces and analyzes two Lie group momentum optimization algorithms, Lie Heavy-Ball and Lie NAG-SC, demonstrating their convergence properties and efficiency advantages on Lie groups.

Contribution

The paper proposes a new Lie NAG-SC optimizer with explicit convergence rates and compares it to existing methods, highlighting computational benefits and acceleration capabilities.

Findings

Lie NAG-SC achieves acceleration over Riemannian gradient descent.

Both methods are computationally cheaper and easier to implement on Lie groups.

Convergence rates are explicitly quantified under smoothness and convexity assumptions.

Abstract

Explicit, momentum-based dynamics that optimize functions defined on Lie groups can be constructed via variational optimization and momentum trivialization. Structure preserving time discretizations can then turn this dynamics into optimization algorithms. This article investigates two types of discretization, Lie Heavy-Ball, which is a known splitting scheme, and Lie NAG-SC, which is newly proposed. Their convergence rates are explicitly quantified under $L$-smoothness and local strong convexity assumptions. Lie NAG-SC provides acceleration over the momentumless case, i.e. Riemannian gradient descent, but Lie Heavy-Ball does not. When compared to existing accelerated optimizers for general manifolds, both Lie Heavy-Ball and Lie NAG-SC are computationally cheaper and easier to implement, thanks to their utilization of group structure. Only gradient oracle and exponential map are required, but not logarithm map or parallel transport which are computational costly.