FedHeN: Federated Learning in Heterogeneous Networks

TL;DR

FedHeN introduces a federated learning approach that enables devices with different architectures to collaboratively train models, improving performance and reducing communication costs in heterogeneous networks.

Contribution

It proposes a novel training recipe with a side objective for higher complexity devices, facilitating joint training of diverse architectures in federated settings.

Findings

Improves performance across different architectures

Achieves significant communication savings

Outperforms state-of-the-art methods

Abstract

We propose a novel training recipe for federated learning with heterogeneous networks where each device can have different architectures. We introduce training with a side objective to the devices of higher complexities to jointly train different architectures in a federated setting. We empirically show that our approach improves the performance of different architectures and leads to high communication savings compared to the state-of-the-art methods.