The Prospect of Enhancing Large-Scale Heterogeneous Federated Learning with Transformers

TL;DR

This paper explores the potential of Transformer-based models in large-scale heterogeneous federated learning, demonstrating their advantages over traditional neural networks through extensive experiments and analysis.

Contribution

It introduces the use of Transformers in federated learning for improved generalization and personalization in large-scale, heterogeneous data scenarios.

Findings

Transformers outperform ResNet in large-scale heterogeneous FL tasks.

Transformers show higher representation similarity across layers, indicating better feature extraction.

Experimental results validate the effectiveness of Transformer-based FL models.

Abstract

Federated learning (FL) addresses data privacy concerns by enabling collaborative training of AI models across distributed data owners. Wide adoption of FL faces the fundamental challenges of data heterogeneity and the large scale of data owners involved. In this paper, we investigate the prospect of Transformer-based FL models for achieving generalization and personalization in this setting. We conduct extensive comparative experiments involving FL with Transformers, ResNet, and personalized ResNet-based FL approaches under various scenarios. These experiments consider varying numbers of data owners to demonstrate Transformers' advantages over deep neural networks in large-scale heterogeneous FL tasks. In addition, we analyze the superior performance of Transformers by comparing the Centered Kernel Alignment (CKA) representation similarity across different layers and FL models to gain insight into the reasons behind their promising capabilities.