Decentralized projected Riemannian stochastic recursive momentum method for nonconvex optimization

TL;DR

This paper introduces a decentralized Riemannian stochastic recursive momentum method for nonconvex optimization on manifolds, achieving improved oracle complexity and efficiency in online settings.

Contribution

It proposes a novel decentralized Riemannian optimization algorithm with superior complexity and practical efficiency for nonconvex problems on manifolds.

Findings

Achieves oracle complexity of (psilon^{-rac{3}{2}})

Requires only (1) gradient evaluations per iteration

Outperforms existing methods in numerical experiments on PCA and matrix completion

Abstract

This paper studies decentralized optimization over a compact submanifold within a communication network of $n$ nodes, where each node possesses a smooth non-convex local cost function, and the goal is to jointly minimize the sum of these local costs. We focus particularly on the online setting, where local data is processed in real-time as it streams in, without the need for full data storage. We propose a decentralized projected Riemannian stochastic recursive momentum (DPRSRM) method that employs local hybrid stochastic gradient estimators and uses the network to track the global gradient. DPRSRM achieves an oracle complexity of \(\mathcal{O}(\epsilon^{-\frac{3}{2}})\), outperforming existing methods that have at most \(\mathcal{O}(\epsilon^{-2})\) complexity. Our method requires only $\mathcal{O}(1)$ gradient evaluations per iteration for each local node and does not require restarting with a large batch gradient. Furthermore, we demonstrate the effectiveness of our proposed methods compared to state-of-the-art ones through numerical experiments on principal component analysis problems and low-rank matrix completion.