Information Entropy-Based Scheduling for Communication-Efficient Decentralized Learning

TL;DR

This paper introduces an information entropy-based scheduling method for decentralized stochastic gradient descent, significantly reducing communication costs while maintaining or improving convergence speed in resource-limited networks.

Contribution

It proposes a novel importance metric based on information entropy for node and link scheduling in decentralized learning, outperforming existing methods like BC and MATCHA.

Findings

Achieves up to 60% lower communication budgets with faster convergence.

Outperforms BC-based node scheduling in efficiency.

Matches or exceeds MATCHA in link scheduling performance.

Abstract

This paper addresses decentralized stochastic gradient descent (D-SGD) over resource-constrained networks by introducing node-based and link-based scheduling strategies to enhance communication efficiency. In each iteration of the D-SGD algorithm, only a few disjoint subsets of nodes or links are randomly activated, subject to a given communication cost constraint. We propose a novel importance metric based on information entropy to determine node and link scheduling probabilities. We validate the effectiveness of our approach through extensive simulations, comparing it against state-of-the-art methods, including betweenness centrality (BC) for node scheduling and \textit{MATCHA} for link scheduling. The results show that our method consistently outperforms the BC-based method in the node scheduling case, achieving faster convergence with up to 60\% lower communication budgets. At higher communication budgets (above 60\%), our method maintains comparable or superior performance. In the link scheduling case, our method delivers results that are superior to or on par with those of \textit{MATCHA}.