A Retraction-Free EXTRA Method for Decentralized Optimization on the Stiefel Manifold

TL;DR

The paper introduces RF-EXTRA, a decentralized optimization algorithm on the Stiefel manifold that avoids retractions, achieves an $ ext{O}(1/K)$ convergence rate, and demonstrates strong communication efficiency in experiments.

Contribution

It proposes a retraction-free primal-dual method for decentralized optimization on the Stiefel manifold with proven convergence guarantees.

Findings

RF-EXTRA converges at an $ ext{O}(1/K)$ rate to a stationary point.

The method exhibits strong communication efficiency in PCA and low-rank matrix completion tasks.

Experiments show RF-EXTRA outperforms existing decentralized baselines.

Abstract

Decentralized optimization provides a fundamental framework for large-scale learning and signal processing with distributed data. We study decentralized optimization with orthogonality constraints on the Stiefel manifold and propose RF-EXTRA, a distributed retraction-free primal-dual method on static undirected networks. The method combines an approximate gradient mapping for orthogonality-constrained optimization with an EXTRA-based decentralized recursion, thereby avoiding retractions while preserving a simple communication pattern. On the theoretical side, the analysis considers \revise{the joint error} $(\mathbf{X}_k-\overline{\mathbf X}_k,\mathbf{s}_k-\overline{\mathbf s}_k)$ in the local variables and local directions, and establishes a contractive recursion for the joint error. This contractivity ensures that the joint error can be controlled using small yet constant step sizes, thus leading to an exact $\mathcal{O}(1/K)$ convergence rate of RF-EXTRA to a stationary point. Experiments on PCA and low-rank matrix completion show that RF-EXTRA compares favorably with the reported decentralized baselines and exhibits strong communication efficiency on the tested tasks on the Stiefel manifold.