Riemannian Anderson Mixing Methods for Minimizing $C^2$-Functions on Riemannian Manifolds

TL;DR

This paper extends Anderson Mixing to Riemannian manifolds, introducing RAM and RRAM methods that improve convergence and efficiency for manifold optimization problems, with proven theoretical properties and superior experimental performance.

Contribution

The paper introduces Riemannian Anderson Mixing (RAM) and its regularized variant (RRAM), providing convergence analysis and demonstrating improved optimization performance on manifolds.

Findings

RAM does not require inverse retraction or exponential mapping.

RRAM guarantees global convergence with local convergence properties.

Experimental results show superior performance over gradient descent and LBFGS.

Abstract

The Anderson Mixing (AM) method is a popular approach for accelerating fixed-point iterations by leveraging historical information from previous steps. In this paper, we introduce the Riemannian Anderson Mixing (RAM) method, an extension of AM to Riemannian manifolds, and analyze its local linear convergence under reasonable assumptions. Unlike other extrapolation-based algorithms on Riemannian manifolds, RAM does not require computing the inverse retraction or inverse exponential mapping and has a lower per-iteration cost. Furthermore, we propose a variant of RAM called Regularized RAM (RRAM), which establishes global convergence and exhibits similar local convergence properties as RAM. Our proof relies on careful error estimations based on the local geometry of Riemannian manifolds. Finally, we present experimental results on various manifold optimization problems that demonstrate the superior performance of our proposed methods over existing Riemannian gradient descent and LBFGS approaches.