Adaptive Guidance for Local Training in Heterogeneous Federated Learning

TL;DR

This paper introduces FedL2G, an adaptive method for guiding local training in heterogeneous federated learning, aligning objectives across diverse models and demonstrating superior performance through extensive experiments.

Contribution

FedL2G is a novel approach that adaptively aligns local training objectives with model heterogeneity using only first-order derivatives, ensuring convergence and improved results.

Findings

FedL2G outperforms seven state-of-the-art methods.

Theoretical guarantees with non-convex convergence rate of O(1/T).

Effective across diverse data and model heterogeneity settings.

Abstract

Model heterogeneity poses a significant challenge in Heterogeneous Federated Learning (HtFL). In scenarios with diverse model architectures, directly aggregating model parameters is impractical, leading HtFL methods to incorporate an extra objective alongside the original local objective on each client to facilitate collaboration. However, this often results in a mismatch between the extra and local objectives. To resolve this, we propose Federated Learning-to-Guide (FedL2G), a method that adaptively learns to guide local training in a federated manner, ensuring the added objective aligns with each client's original goal. With theoretical guarantees, FedL2G utilizes only first-order derivatives w.r.t. model parameters, achieving a non-convex convergence rate of O(1/T). We conduct extensive experiments across two data heterogeneity and six model heterogeneity settings, using 14 heterogeneous model architectures (e.g., CNNs and ViTs). The results show that FedL2G significantly outperforms seven state-of-the-art methods.