Communication-Efficient {Federated} Learning Using Censored Heavy Ball Descent

TL;DR

This paper introduces a censoring-based heavy ball method for federated learning that reduces communication by only transmitting significant gradient updates, maintaining convergence speed and accuracy.

Contribution

It proposes a novel CHB algorithm that adaptively censors small updates, reducing communication without sacrificing convergence in both convex and nonconvex settings.

Findings

Achieves linear convergence rate similar to classical heavy ball method.

Can eliminate at least half of communications without affecting convergence.

Validates effectiveness on synthetic and real datasets across various problem types.

Abstract

Distributed machine learning enables scalability and computational offloading, but requires significant levels of communication. Consequently, communication efficiency in distributed learning settings is an important consideration, especially when the communications are wireless and battery-driven devices are employed. In this paper we develop a censoring-based heavy ball (CHB) method for distributed learning in a server-worker architecture. Each worker self-censors unless its local gradient is sufficiently different from the previously transmitted one. The significant practical advantages of the HB method for learning problems are well known, but the question of reducing communications has not been addressed. CHB takes advantage of the HB smoothing to eliminate reporting small changes, and provably achieves a linear convergence rate equivalent to that of the classical HB method for smooth and strongly convex objective functions. The convergence guarantee of CHB is theoretically justified for both convex and nonconvex cases. In addition we prove that, under some conditions, at least half of all communications can be eliminated without any impact on convergence rate. Extensive numerical results validate the communication efficiency of CHB on both synthetic and real datasets, for convex, nonconvex, and nondifferentiable cases. Given a target accuracy, CHB can significantly reduce the number of communications compared to existing algorithms, achieving the same accuracy without slowing down the optimization process.