Communication-Efficient Distributed SVD via Local Power Iterations

TL;DR

This paper introduces exttt{LocalPower}, a communication-efficient distributed SVD algorithm that combines local power iterations with global aggregation, significantly reducing communication costs while maintaining accuracy.

Contribution

The paper proposes exttt{LocalPower}, a novel distributed SVD method that reduces communication by performing multiple local power iterations before global aggregation, with theoretical and experimental validation.

Findings

exttt{LocalPower} reduces communication by a factor of p.

Sign-fixing improves stability and efficiency.

Decaying p enhances high-precision solutions.

Abstract

We study distributed computing of the truncated singular value decomposition problem. We develop an algorithm that we call \texttt{LocalPower} for improving communication efficiency. Specifically, we uniformly partition the dataset among $m$ nodes and alternate between multiple (precisely $p$) local power iterations and one global aggregation. In the aggregation, we propose to weight each local eigenvector matrix with orthogonal Procrustes transformation (OPT). As a practical surrogate of OPT, sign-fixing, which uses a diagonal matrix with $\pm 1$ entries as weights, has better computation complexity and stability in experiments. We theoretically show that under certain assumptions \texttt{LocalPower} lowers the required number of communications by a factor of $p$ to reach a constant accuracy. We also show that the strategy of periodically decaying $p$ helps obtain high-precision solutions. We conduct experiments to demonstrate the effectiveness of \texttt{LocalPower}.