Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning

TL;DR

This paper introduces FedFD, a feature distillation approach for model-heterogeneous federated learning, which aligns features across diverse models using orthogonal projections to improve knowledge aggregation and model performance.

Contribution

The paper proposes a novel feature distillation method with orthogonal projections to better handle heterogeneity in federated learning models, enhancing stability and accuracy.

Findings

FedFD outperforms existing methods in experiments.

Orthogonal projection mitigates knowledge bias from heterogeneous models.

Feature-based distillation improves global model performance.

Abstract

Model-Heterogeneous Federated Learning (Hetero-FL) has attracted growing attention for its ability to aggregate knowledge from heterogeneous models while keeping private data locally. To better aggregate knowledge from clients, ensemble distillation, as a widely used and effective technique, is often employed after global aggregation to enhance the performance of the global model. However, simply combining Hetero-FL and ensemble distillation does not always yield promising results and can make the training process unstable. The reason is that existing methods primarily focus on logit distillation, which, while being model-agnostic with softmax predictions, fails to compensate for the knowledge bias arising from heterogeneous models. To tackle this challenge, we propose a stable and efficient Feature Distillation for model-heterogeneous Federated learning, dubbed FedFD, that can incorporate aligned feature information via orthogonal projection to integrate knowledge from heterogeneous models better. Specifically, a new feature-based ensemble federated knowledge distillation paradigm is proposed. The global model on the server needs to maintain a projection layer for each client-side model architecture to align the features separately. Orthogonal techniques are employed to re-parameterize the projection layer to mitigate knowledge bias from heterogeneous models and thus maximize the distilled knowledge. Extensive experiments show that FedFD achieves superior performance compared to state-of-the-art methods.