No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices

TL;DR

InclusiveFL enables federated learning across heterogeneous devices by assigning different model sizes to clients based on their capabilities and sharing knowledge among models, ensuring all clients participate effectively.

Contribution

This paper introduces InclusiveFL, a novel federated learning approach that supports heterogeneous devices by using multi-sized models and knowledge sharing, improving inclusivity and accuracy.

Findings

Effective in training accurate models on heterogeneous devices

Outperforms existing FL methods in real-world benchmarks

Maintains high participation of weak clients

Abstract

Federated learning (FL) is an important paradigm for training global models from decentralized data in a privacy-preserving way. Existing FL methods usually assume the global model can be trained on any participating client. However, in real applications, the devices of clients are usually heterogeneous, and have different computing power. Although big models like BERT have achieved huge success in AI, it is difficult to apply them to heterogeneous FL with weak clients. The straightforward solutions like removing the weak clients or using a small model to fit all clients will lead to some problems, such as under-representation of dropped clients and inferior accuracy due to data loss or limited model representation ability. In this work, we propose InclusiveFL, a client-inclusive federated learning method to handle this problem. The core idea of InclusiveFL is to assign models of different sizes to clients with different computing capabilities, bigger models for powerful clients and smaller ones for weak clients. We also propose an effective method to share the knowledge among multiple local models with different sizes. In this way, all the clients can participate in the model learning in FL, and the final model can be big and powerful enough. Besides, we propose a momentum knowledge distillation method to better transfer knowledge in big models on powerful clients to the small models on weak clients. Extensive experiments on many real-world benchmark datasets demonstrate the effectiveness of the proposed method in learning accurate models from clients with heterogeneous devices under the FL framework.