Weight Divergence Driven Divide-and-Conquer Approach for Optimal Federated Learning from non-IID Data

TL;DR

This paper introduces a divide-and-conquer federated learning approach that uses a novel weight divergence metric to effectively handle non-IID data, improving accuracy and resource efficiency.

Contribution

It presents a new methodology leveraging cosine-distance based weight divergence to split neural networks for better federated learning on non-IID data.

Findings

Achieves accuracy comparable or superior to FedProx and FedMA.

Optimizes compute and bandwidth usage under specific conditions.

Enables effective use of FedAvg in non-IID environments.

Abstract

Federated Learning allows training of data stored in distributed devices without the need for centralizing training data, thereby maintaining data privacy. Addressing the ability to handle data heterogeneity (non-identical and independent distribution or non-IID) is a key enabler for the wider deployment of Federated Learning. In this paper, we propose a novel Divide-and-Conquer training methodology that enables the use of the popular FedAvg aggregation algorithm by overcoming the acknowledged FedAvg limitations in non-IID environments. We propose a novel use of Cosine-distance based Weight Divergence metric to determine the exact point where a Deep Learning network can be divided into class agnostic initial layers and class-specific deep layers for performing a Divide and Conquer training. We show that the methodology achieves trained model accuracy at par (and in certain cases exceeding) with numbers achieved by state-of-the-art Aggregation algorithms like FedProx, FedMA, etc. Also, we show that this methodology leads to compute and bandwidth optimizations under certain documented conditions.