Federated Learning for Inference at Anytime and Anywhere

TL;DR

This paper introduces a novel federated learning framework that adapts pre-trained Transformer models using attention-based adapters, enabling efficient, personalized, and scalable inference across heterogeneous devices.

Contribution

Proposes a new method to adapt pre-trained Transformers in federated learning using attention-based adapters, improving efficiency, personalization, and scalability.

Findings

Fast and communication-efficient training with heterogeneous data.

Supports diverse device capabilities for inference.

Achieves accurate and scalable federated learning results.

Abstract

Federated learning has been predominantly concerned with collaborative training of deep networks from scratch, and especially the many challenges that arise, such as communication cost, robustness to heterogeneous data, and support for diverse device capabilities. However, there is no unified framework that addresses all these problems together. This paper studies the challenges and opportunities of exploiting pre-trained Transformer models in FL. In particular, we propose to efficiently adapt such pre-trained models by injecting a novel attention-based adapter module at each transformer block that both modulates the forward pass and makes an early prediction. Training only the lightweight adapter by FL leads to fast and communication-efficient learning even in the presence of heterogeneous data and devices. Extensive experiments on standard FL benchmarks, including CIFAR-100, FEMNIST and SpeechCommandsv2 demonstrate that this simple framework provides fast and accurate FL while supporting heterogenous device capabilities, efficient personalization, and scalable-cost anytime inference.