Exploiting Similarity for Computation and Communication-Efficient Decentralized Optimization

TL;DR

This paper introduces SPDO, a new decentralized optimization method that reduces both communication and computational costs by exploiting functional similarity among nodes, with theoretical guarantees and superior experimental performance.

Contribution

The paper proposes SPDO, a stabilized PDO method that achieves optimal complexities and relaxes subproblem accuracy requirements by leveraging average similarity.

Findings

SPDO outperforms existing PDO methods in experiments.

SPDO achieves state-of-the-art communication and computational efficiencies.

Relaxed subproblem accuracy requirements improve practical performance.

Abstract

Reducing communication complexity is critical for efficient decentralized optimization. The proximal decentralized optimization (PDO) framework is particularly appealing, as methods within this framework can exploit functional similarity among nodes to reduce communication rounds. Specifically, when local functions at different nodes are similar, these methods achieve faster convergence with fewer communication steps. However, existing PDO methods often require highly accurate solutions to subproblems associated with the proximal operator, resulting in significant computational overhead. In this work, we propose the Stabilized Proximal Decentralized Optimization (SPDO) method, which achieves state-of-the-art communication and computational complexities within the PDO framework. Additionally, we refine the analysis of existing PDO methods by relaxing subproblem accuracy requirements and leveraging average functional similarity. Experimental results demonstrate that SPDO significantly outperforms existing methods.