Revisiting the Primal-Dual Method of Multipliers for Optimisation over Centralised Networks

TL;DR

This paper revisits the primal-dual method of multipliers for centralized network optimization, proposing improved algorithms with tighter convergence bounds and analyzing their performance relative to existing methods.

Contribution

It introduces two enhanced versions of Inexact PDMM, GPDMM and AGPDMM, with theoretical convergence guarantees and empirical performance analysis in federated optimization.

Findings

AGPDMM accelerates convergence compared to GPDMM.

GPDMM has tighter convergence bounds than Inexact FedSplit.

AGPDMM outperforms SCAFFOLD when multiple gradient steps are used.

Abstract

The primal-dual method of multipliers (PDMM) was originally designed for solving a decomposable optimisation problem over a general network. In this paper, we revisit PDMM for optimisation over a centralized network. We first note that the recently proposed method FedSplit [1] implements PDMM for a centralized network. In [1], Inexact FedSplit (i.e., gradient based FedSplit) was also studied both empirically and theoretically. We identify the cause for the poor reported performance of Inexact FedSplit, which is due to the improper initialisation in the gradient operations at the client side. To fix the issue of Inexact FedSplit, we propose two versions of Inexact PDMM, which are referred to as gradient-based PDMM (GPDMM) and accelerated GPDMM (AGPDMM), respectively. AGPDMM accelerates GPDMM at the cost of transmitting two times the number of parameters from the server to each client per iteration compared to GPDMM. We provide a new convergence bound for GPDMM for a class of convex optimisation problems. Our new bounds are tighter than those derived for Inexact FedSplit. We also investigate the update expressions of AGPDMM and SCAFFOLD to find their similarities. It is found that when the number K of gradient steps at the client side per iteration is K=1, both AGPDMM and SCAFFOLD reduce to vanilla gradient descent with proper parameter setup. Experimental results indicate that AGPDMM converges faster than SCAFFOLD when K>1 while GPDMM converges slightly worse than SCAFFOLD.