Feature-Aware One-Shot Federated Learning via Hierarchical Token Sequences

TL;DR

FALCON introduces a feature-aware hierarchical token sequence approach combined with knowledge distillation to enhance one-shot federated learning on non-IID image data, achieving significant accuracy improvements in medical and natural image tasks.

Contribution

The paper proposes FALCON, a novel framework that leverages hierarchical token sequences and knowledge distillation to improve one-shot federated learning on non-IID data.

Findings

Outperforms existing OSFL methods by 9.58% in average accuracy.

Effectively handles non-IID data in medical and natural images.

Demonstrates robustness and efficiency in real-world scenarios.

Abstract

One-shot federated learning (OSFL) reduces the communication cost and privacy risks of iterative federated learning by constructing a global model with a single round of communication. However, most existing methods struggle to achieve robust performance on real-world domains such as medical imaging, or are inefficient when handling non-IID (Independent and Identically Distributed) data. To address these limitations, we introduce FALCON, a framework that enhances the effectiveness of OSFL over non-IID image data. The core idea of FALCON is to leverage the feature-aware hierarchical token sequences generation and knowledge distillation into OSFL. First, each client leverages a pretrained visual encoder with hierarchical scale encoding to compress images into hierarchical token sequences, which capture multi-scale semantics. Second, a multi-scale autoregressive transformer generator is used to model the distribution of these token sequences and generate the synthetic sequences. Third, clients upload the synthetic sequences along with the local classifier trained on the real token sequences to the server. Finally, the server incorporates knowledge distillation into global training to reduce reliance on precise distribution modeling. Experiments on medical and natural image datasets validate the effectiveness of FALCON in diverse non-IID scenarios, outperforming the best OSFL baselines by 9.58% in average accuracy.