Distributed Retraction-Free and Communication-Efficient Optimization on the Stiefel Manifold

TL;DR

This paper introduces EF-Landing, a novel distributed optimization algorithm on the Stiefel manifold that is retraction-free, communication-efficient, and guarantees convergence with reduced communication overhead.

Contribution

It extends the Landing algorithm to a distributed setting with communication compression and error feedback, providing the first retraction-free, communication-efficient method for this problem.

Findings

Achieves the same asymptotic linear speedup as existing methods

Guarantees convergence and constraint feasibility with compressed communication

Applicable to both deterministic and stochastic settings, including gradient and momentum methods

Abstract

Optimization problems on the Stiefel manifold, ranging from principal component analysis to enhancing neural network robustness, are ubiquitous in machine learning. The Landing algorithm avoids computationally expensive retraction operations on manifolds, making it highly competitive for large-scale problems. This paper extends this method to distributed settings, introducing *EF-Landing*, the first retraction-free and communication-efficient algorithm for distributed stochastic optimization on the Stiefel manifold. By incorporating communication compression and error feedback, EF-Landing ensures convergence and constraint feasibility while significantly reducing communication overhead. We provide sharp convergence guarantees, demonstrating that EF-Landing achieves the same asymptotic linear speedup convergence rate as existing methods without communication compression. Furthermore, our analysis is highly versatile, applying to both deterministic and stochastic settings and encompassing algorithms based on gradient descent or momentum-based gradient descent. We also generalize EF-Landing to operate on block-wise Stiefel manifolds, enabling greater flexibility for structured constraints. Extensive numerical experiments validate our theoretical results.