Handling Data Heterogeneity via Architectural Design for Federated Visual Recognition

TL;DR

This paper investigates how architectural design choices in neural networks impact federated learning performance for visual recognition, emphasizing the importance of model architecture in handling data heterogeneity.

Contribution

It provides an extensive analysis of various neural network architectures in federated visual recognition, highlighting the significance of design choices for improved performance.

Findings

Architectural choices significantly influence federated learning performance.

Transformers and MLP-mixers outperform shallow networks in FL settings.

Normalization layers impact model performance in federated visual recognition.

Abstract

Federated Learning (FL) is a promising research paradigm that enables the collaborative training of machine learning models among various parties without the need for sensitive information exchange. Nonetheless, retaining data in individual clients introduces fundamental challenges to achieving performance on par with centrally trained models. Our study provides an extensive review of federated learning applied to visual recognition. It underscores the critical role of thoughtful architectural design choices in achieving optimal performance, a factor often neglected in the FL literature. Many existing FL solutions are tested on shallow or simple networks, which may not accurately reflect real-world applications. This practice restricts the transferability of research findings to large-scale visual recognition models. Through an in-depth analysis of diverse cutting-edge architectures such as convolutional neural networks, transformers, and MLP-mixers, we experimentally demonstrate that architectural choices can substantially enhance FL systems' performance, particularly when handling heterogeneous data. We study 19 visual recognition models from five different architectural families on four challenging FL datasets. We also re-investigate the inferior performance of convolution-based architectures in the FL setting and analyze the influence of normalization layers on the FL performance. Our findings emphasize the importance of architectural design for computer vision tasks in practical scenarios, effectively narrowing the performance gap between federated and centralized learning. Our source code is available at https://github.com/sarapieri/fed_het.git.