Sporadic Gradient Tracking over Directed Graphs: A Theoretical Perspective on Decentralized Federated Learning

TL;DR

This paper introduces Spod-GT, a novel decentralized federated learning algorithm that handles client heterogeneity and resource variability over directed graphs, with proven convergence and superior experimental performance.

Contribution

It is the first to unify gradient tracking with sporadic client participation and resource heterogeneity over directed graphs in decentralized federated learning.

Findings

Spod-GT converges under relaxed assumptions.

It outperforms existing gradient tracking methods.

Handles asymmetric and intermittent client communication.

Abstract

Decentralized Federated Learning (DFL) enables clients with local data to collaborate in a peer-to-peer manner to train a generalized model. In this paper, we unify two branches of work that have separately solved important challenges in DFL: (i) gradient tracking techniques for mitigating data heterogeneity and (ii) accounting for diverse availability of resources across clients. We propose $\textit{Sporadic Gradient Tracking}$ ($\texttt{Spod-GT}$), the first DFL algorithm that incorporates these factors over general directed graphs by allowing (i) client-specific gradient computation frequencies and (ii) heterogeneous and asymmetric communication frequencies. We conduct a rigorous convergence analysis of our methodology with relaxed assumptions on gradient estimation variance and gradient diversity of clients, providing consensus and optimality guarantees for GT over directed graphs despite intermittent client participation. Through numerical experiments on image classification datasets, we demonstrate the efficacy of $\texttt{Spod-GT}$ compared to well-known GT baselines.