FedNNNN: Norm-Normalized Neural Network Aggregation for Fast and Accurate Federated Learning

TL;DR

FedNNNN introduces a norm-normalized aggregation method for federated learning, enhancing convergence speed and accuracy over traditional averaging techniques like FedAvg by adjusting update vector norms and employing momentum control.

Contribution

The paper proposes FedNNNN, a novel aggregation method that normalizes update vector norms and uses momentum control to improve federated learning performance.

Findings

Up to 5.4% accuracy improvement over FedAvg

Faster convergence in federated learning scenarios

Effective across multiple datasets and neural network models

Abstract

Federated learning (FL) is a distributed learning protocol in which a server needs to aggregate a set of models learned some independent clients to proceed the learning process. At present, model averaging, known as FedAvg, is one of the most widely adapted aggregation techniques. However, it is known to yield the models with degraded prediction accuracy and slow convergence. In this work, we find out that averaging models from different clients significantly diminishes the norm of the update vectors, resulting in slow learning rate and low prediction accuracy. Therefore, we propose a new aggregation method called FedNNNN. Instead of simple model averaging, we adjust the norm of the update vector and introduce momentum control techniques to improve the aggregation effectiveness of FL. As a demonstration, we evaluate FedNNNN on multiple datasets and scenarios with different neural network models, and observe up to 5.4% accuracy improvement.