A Variance-Reduced Stochastic Gradient Tracking Algorithm for Decentralized Optimization with Orthogonality Constraints

TL;DR

This paper introduces VRSGT, a novel decentralized optimization algorithm that efficiently handles orthogonality constraints by reducing both sampling and communication complexities, with promising real-world application results.

Contribution

The paper presents the first variance-reduced stochastic gradient tracking algorithm for decentralized optimization with orthogonality constraints that reduces both sample and communication complexities.

Findings

VRSGT achieves an $O(1/k)$ convergence rate.

VRSGT reduces both sampling and communication complexities.

VRSGT performs well in autonomous driving applications.

Abstract

Decentralized optimization with orthogonality constraints is found widely in scientific computing and data science. Since the orthogonality constraints are nonconvex, it is quite challenging to design efficient algorithms. Existing approaches leverage the geometric tools from Riemannian optimization to solve this problem at the cost of high sample and communication complexities. To relieve this difficulty, based on two novel techniques that can waive the orthogonality constraints, we propose a variance-reduced stochastic gradient tracking (VRSGT) algorithm with the convergence rate of $O(1 / k)$ to a stationary point. To the best of our knowledge, VRSGT is the first algorithm for decentralized optimization with orthogonality constraints that reduces both sampling and communication complexities simultaneously. In the numerical experiments, VRSGT has a promising performance in a real-world autonomous driving application.