Stabilized Proximal-Point Methods for Federated Optimization

TL;DR

This paper introduces S-DANE, a novel federated optimization algorithm that improves communication and local computation efficiency, supports partial participation, and is adaptable with provable guarantees.

Contribution

It proposes S-DANE, an improved distributed proximal-point method that reduces local subproblem accuracy requirements and accelerates communication efficiency in federated learning.

Findings

S-DANE achieves the best-known communication complexity.

The algorithm supports partial client participation.

Adaptive variants with line search are provably efficient.

Abstract

In developing efficient optimization algorithms, it is crucial to account for communication constraints -- a significant challenge in modern Federated Learning. The best-known communication complexity among non-accelerated algorithms is achieved by DANE, a distributed proximal-point algorithm that solves local subproblems at each iteration and that can exploit second-order similarity among individual functions. However, to achieve such communication efficiency, the algorithm requires solving local subproblems sufficiently accurately resulting in slightly sub-optimal local complexity. Inspired by the hybrid-projection proximal-point method, in this work, we propose a novel distributed algorithm S-DANE. Compared to DANE, this method uses an auxiliary sequence of prox-centers while maintaining the same deterministic communication complexity. Moreover, the accuracy condition for solving the subproblem is milder, leading to enhanced local computation efficiency. Furthermore, S-DANE supports partial client participation and arbitrary stochastic local solvers, making it attractive in practice. We further accelerate S-DANE and show that the resulting algorithm achieves the best-known communication complexity among all existing methods for distributed convex optimization while still enjoying good local computation efficiency as S-DANE. Finally, we propose adaptive variants of both methods using line search, obtaining the first provably efficient adaptive algorithms that could exploit local second-order similarity without the prior knowledge of any parameters.