Layer-wise Adaptive Model Aggregation for Scalable Federated Learning

TL;DR

FedLAMA introduces a layer-wise adaptive aggregation method in federated learning that reduces communication costs significantly while maintaining model accuracy, by considering layer-specific discrepancies and adjusting aggregation intervals accordingly.

Contribution

The paper presents FedLAMA, a novel layer-wise adaptive aggregation scheme that improves scalability and communication efficiency in federated learning.

Findings

Reduces communication cost by up to 60% for IID data.

Reduces communication cost by up to 70% for non-IID data.

Achieves comparable accuracy to FedAvg.

Abstract

In Federated Learning, a common approach for aggregating local models across clients is periodic averaging of the full model parameters. It is, however, known that different layers of neural networks can have a different degree of model discrepancy across the clients. The conventional full aggregation scheme does not consider such a difference and synchronizes the whole model parameters at once, resulting in inefficient network bandwidth consumption. Aggregating the parameters that are similar across the clients does not make meaningful training progress while increasing the communication cost. We propose FedLAMA, a layer-wise model aggregation scheme for scalable Federated Learning. FedLAMA adaptively adjusts the aggregation interval in a layer-wise manner, jointly considering the model discrepancy and the communication cost. The layer-wise aggregation method enables to finely control the aggregation interval to relax the aggregation frequency without a significant impact on the model accuracy. Our empirical study shows that FedLAMA reduces the communication cost by up to 60% for IID data and 70% for non-IID data while achieving a comparable accuracy to FedAvg.