Federated Cross-Training Learners for Robust Generalization under Data Heterogeneity

TL;DR

This paper introduces FedCT, a federated learning scheme that uses multi-view knowledge distillation, feature augmentation, and consistency mechanisms to improve model generalization across heterogeneous data sources.

Contribution

FedCT proposes a novel cross-training framework with modules for knowledge broadcasting, multi-view representation learning, and feature augmentation to enhance federated learning under data heterogeneity.

Findings

FedCT outperforms state-of-the-art methods on four datasets.

It effectively alleviates knowledge forgetting in federated learning.

The approach improves model generalization and robustness.

Abstract

Federated learning benefits from cross-training strategies, which enables models to train on data from distinct sources to improve generalization capability. However, due to inherent differences in data distributions, the optimization goals of local models remain misaligned, and this mismatch continues to manifest as feature space heterogeneity even after cross-training. We argue that knowledge distillation from the personalized view preserves client-specific characteristics and expands the local knowledge base, while distillation from the global view provides consistent semantic anchors that facilitate feature alignment across clients. To achieve this goal, this paper presents a cross-training scheme, termed FedCT, includes three main modules, where the consistency-aware knowledge broadcasting module aims to optimize model assignment strategies, which enhances collaborative advantages between clients and achieves an efficient federated learning process. The multi-view knowledge-guided representation learning module leverages fused prototypical knowledge from both global and local views to enhance the preservation of local knowledge before and after model exchange, as well as to ensure consistency between local and global knowledge. The mixup-based feature augmentation module aggregates rich information to further increase the diversity of feature spaces, which enables the model to better discriminate complex samples. Extensive experiments were conducted on four datasets in terms of performance comparison, ablation study, in-depth analysis and case study. The results demonstrated that FedCT alleviates knowledge forgetting from both local and global views, which enables it outperform state-of-the-art methods.