Towards efficient compression and communication for prototype-based decentralized learning

TL;DR

This paper proposes a communication-efficient decentralized prototype-based learning system that reduces data transmission through selective updates and clustering, maintaining convergence speed while improving robustness and adaptability.

Contribution

It introduces a novel twofold data compression approach and parallel gossiping in decentralized prototype learning, enhancing efficiency and robustness.

Findings

Significant reduction in communication load without affecting convergence.

Effective prototype clustering improves message compression.

Parallel gossiping reduces age-of-information, enhancing responsiveness.

Abstract

In prototype-based federated learning, the exchange of model parameters between clients and the master server is replaced by transmission of prototypes or quantized versions of the data samples to the aggregation server. A fully decentralized deployment of prototype-based learning, without a central agregartor of prototypes, is more robust upon network failures and reacts faster to changes in the statistical distribution of the data, suggesting potential advantages and quick adaptation in dynamic learning tasks, e.g., when the data sources are IoT devices or when data is non-iid. In this paper, we consider the problem of designing a communication-efficient decentralized learning system based on prototypes. We address the challenge of prototype redundancy by leveraging on a twofold data compression technique, i.e., sending only update messages if the prototypes are informationtheoretically useful (via the Jensen-Shannon distance), and using clustering on the prototypes to compress the update messages used in the gossip protocol. We also use parallel instead of sequential gossiping, and present an analysis of its age-of-information (AoI). Our experimental results show that, with these improvements, the communications load can be substantially reduced without decreasing the convergence rate of the learning algorithm.