Robust Model Aggregation for Heterogeneous Federated Learning: Analysis and Optimizations

TL;DR

This paper introduces a time-driven synchronous federated learning framework tailored for heterogeneous systems, optimizing model aggregation and selection to enhance robustness, reduce latency, and improve accuracy compared to existing asynchronous methods.

Contribution

The paper proposes a novel T-SFL framework with an optimized aggregation scheme and a discriminative model selection algorithm, addressing heterogeneity challenges in federated learning.

Findings

Reduces latency of federated learning by 50%.

Achieves 3% higher accuracy than state-of-the-art AFL algorithms.

Provides theoretical upper bounds on global loss for T-SFL.

Abstract

Conventional synchronous federated learning (SFL) frameworks suffer from performance degradation in heterogeneous systems due to imbalanced local data size and diverse computing power on the client side. To address this problem, asynchronous FL (AFL) and semi-asynchronous FL have been proposed to recover the performance loss by allowing asynchronous aggregation. However, asynchronous aggregation incurs a new problem of inconsistency between local updates and global updates. Motivated by the issues of conventional SFL and AFL, we first propose a time-driven SFL (T-SFL) framework for heterogeneous systems. The core idea of T-SFL is that the server aggregates the models from different clients, each with varying numbers of iterations, at regular time intervals. To evaluate the learning performance of T-SFL, we provide an upper bound on the global loss function. Further, we optimize the aggregation weights to minimize the developed upper bound. Then, we develop a discriminative model selection (DMS) algorithm that removes local models from clients whose number of iterations falls below a predetermined threshold. In particular, this algorithm ensures that each client's aggregation weight accurately reflects its true contribution to the global model update, thereby improving the efficiency and robustness of the system. To validate the effectiveness of T-SFL with the DMS algorithm, we conduct extensive experiments using several popular datasets including MNIST, Cifar-10, Fashion-MNIST, and SVHN. The experimental results demonstrate that T-SFL with the DMS algorithm can reduce the latency of conventional SFL by 50\%, while achieving an average 3\% improvement in learning accuracy over state-of-the-art AFL algorithms.