SignSGD with Federated Voting

TL;DR

This paper introduces signSGD-FV, a federated voting-based optimizer for distributed learning that effectively handles heterogeneous mini-batch sizes, reducing communication costs and improving convergence rates.

Contribution

It proposes a novel signSGD optimizer with federated voting that adaptively weights edge devices, providing convergence guarantees under heterogeneity and demonstrating superior performance.

Findings

SignSGD-FV converges faster than signSGD-MV in heterogeneous settings.

Federated voting improves gradient sign decoding accuracy.

Theoretical analysis confirms convergence with imperfect weight estimation.

Abstract

Distributed learning is commonly used for accelerating model training by harnessing the computational capabilities of multiple-edge devices. However, in practical applications, the communication delay emerges as a bottleneck due to the substantial information exchange required between workers and a central parameter server. SignSGD with majority voting (signSGD-MV) is an effective distributed learning algorithm that can significantly reduce communication costs by one-bit quantization. However, due to heterogeneous computational capabilities, it fails to converge when the mini-batch sizes differ among workers. To overcome this, we propose a novel signSGD optimizer with \textit{federated voting} (signSGD-FV). The idea of federated voting is to exploit learnable weights to perform weighted majority voting. The server learns the weights assigned to the edge devices in an online fashion based on their computational capabilities. Subsequently, these weights are employed to decode the signs of the aggregated local gradients in such a way to minimize the sign decoding error probability. We provide a unified convergence rate analysis framework applicable to scenarios where the estimated weights are known to the parameter server either perfectly or imperfectly. We demonstrate that the proposed signSGD-FV algorithm has a theoretical convergence guarantee even when edge devices use heterogeneous mini-batch sizes. Experimental results show that signSGD-FV outperforms signSGD-MV, exhibiting a faster convergence rate, especially in heterogeneous mini-batch sizes.