Communication-Efficient Local SGD with Age-Based Worker Selection

TL;DR

This paper introduces AgeSel, an age-based worker selection strategy for distributed local SGD that improves communication efficiency and convergence speed by balancing worker participation based on their ages.

Contribution

The paper proposes a novel age-based worker selection method, AgeSel, which enhances communication efficiency and convergence in distributed local SGD with heterogeneous data.

Findings

AgeSel reduces training rounds to reach target accuracy.

It significantly cuts communication costs.

The hyper-parameter influences the effectiveness of worker selection.

Abstract

A major bottleneck of distributed learning under parameter-server (PS) framework is communication cost due to frequent bidirectional transmissions between the PS and workers. To address this issue, local stochastic gradient descent (SGD) and worker selection have been exploited by reducing the communication frequency and the number of participating workers at each round, respectively. However, partial participation can be detrimental to convergence rate, especially for heterogeneous local datasets. In this paper, to improve communication efficiency and speed up the training process, we develop a novel worker selection strategy named AgeSel. The key enabler of AgeSel is utilization of the ages of workers to balance their participation frequencies. The convergence of local SGD with the proposed age-based partial worker participation is rigorously established. Simulation results demonstrate that the proposed AgeSel strategy can significantly reduce the number of training rounds needed to achieve a targeted accuracy, as well as the communication cost. The influence of the algorithm hyper-parameter is also explored to manifest the benefit of age-based worker selection.