A Communication-Efficient Adaptive Algorithm for Federated Learning under Cumulative Regret

TL;DR

This paper introduces a distributed online learning algorithm for federated settings that achieves near-optimal cumulative regret while significantly reducing communication costs, addressing both efficiency and communication challenges.

Contribution

It presents a novel communication-efficient adaptive algorithm for federated learning that optimizes cumulative regret with low total communication overhead.

Findings

Achieves order-optimal cumulative regret in distributed online learning.

Reduces total communication bits compared to existing methods.

Balances learning accuracy with communication efficiency.

Abstract

We consider the problem of online stochastic optimization in a distributed setting with $M$ clients connected through a central server. We develop a distributed online learning algorithm that achieves order-optimal cumulative regret with low communication cost measured in the total number of bits transmitted over the entire learning horizon. This is in contrast to existing studies which focus on the offline measure of simple regret for learning efficiency. The holistic measure for communication cost also departs from the prevailing approach that \emph{separately} tackles the communication frequency and the number of bits in each communication round.