Delta Sum Learning: an approach for fast and global convergence in Gossip Learning

TL;DR

Delta Sum Learning enhances Gossip Learning by improving convergence speed and accuracy in decentralized federated settings, especially at larger scales, through a novel aggregation method and a Kubernetes-based deployment framework.

Contribution

The paper introduces Delta Sum Learning, a new aggregation technique for Gossip Learning, and implements it within a declarative, decentralized orchestration framework for scalable edge deployment.

Findings

Delta Sum achieves comparable performance to existing methods in 10-node topologies.

It reduces global accuracy drop by 58% when scaling to 50 nodes.

It demonstrates logarithmic accuracy loss with increasing topology size.

Abstract

Federated Learning is a popular approach for distributed learning due to its security and computational benefits. With the advent of powerful devices in the network edge, Gossip Learning further decentralizes Federated Learning by removing centralized integration and relying fully on peer to peer updates. However, the averaging methods generally used in both Federated and Gossip Learning are not ideal for model accuracy and global convergence. Additionally, there are few options to deploy Learning workloads in the edge as part of a larger application using a declarative approach such as Kubernetes manifests. This paper proposes Delta Sum Learning as a method to improve the basic aggregation operation in Gossip Learning, and implements it in a decentralized orchestration framework based on Open Application Model, which allows for dynamic node discovery and intent-driven deployment of multi-workload applications. Evaluation results show that Delta Sum performance is on par with alternative integration methods for 10 node topologies, but results in a 58% lower global accuracy drop when scaling to 50 nodes. Overall, it shows strong global convergence and a logarithmic loss of accuracy with increasing topology size compared to a linear loss for alternatives under limited connectivity.