TornadoAggregate: Accurate and Scalable Federated Learning via the Ring-Based Architecture

TL;DR

TornadoAggregate introduces a ring-based federated learning algorithm that enhances accuracy and scalability by reducing variance through innovative grouping and chaining strategies, addressing communication and diurnal property challenges.

Contribution

It presents a novel variance reduction approach within ring architecture for federated learning, improving accuracy and scalability over prior star-topology methods.

Findings

Test accuracy improved by up to 26.7%.

Achieved near-linear scalability.

Effectively reduces variance in ring-based federated learning.

Abstract

Federated learning has emerged as a new paradigm of collaborative machine learning; however, many prior studies have used global aggregation along a star topology without much consideration of the communication scalability or the diurnal property relied on clients' local time variety. In contrast, ring architecture can resolve the scalability issue and even satisfy the diurnal property by iterating nodes without an aggregation. Nevertheless, such ring-based algorithms can inherently suffer from the high-variance problem. To this end, we propose a novel algorithm called TornadoAggregate that improves both accuracy and scalability by facilitating the ring architecture. In particular, to improve the accuracy, we reformulate the loss minimization into a variance reduction problem and establish three principles to reduce variance: Ring-Aware Grouping, Small Ring, and Ring Chaining. Experimental results show that TornadoAggregate improved the test accuracy by up to 26.7% and achieved near-linear scalability.