FedBone: Towards Large-Scale Federated Multi-Task Learning

TL;DR

FedBone introduces a novel federated multi-task learning framework that leverages server-client split learning and gradient projection to enable large-scale models on edge devices, improving task generalization and performance.

Contribution

The paper proposes FedBone, a new framework that splits large models between server and clients and uses gradient projection to handle heterogeneity and enable large-scale federated multi-task learning.

Findings

Outperforms existing federated learning algorithms on benchmark datasets.

Effectively handles heterogeneous local tasks with off-the-shelf resources.

Enhances generalization of large models through gradient conflict mitigation.

Abstract

Heterogeneous federated multi-task learning (HFMTL) is a federated learning technique that combines heterogeneous tasks of different clients to achieve more accurate, comprehensive predictions. In real-world applications, visual and natural language tasks typically require large-scale models to extract high-level abstract features. However, large-scale models cannot be directly applied to existing federated multi-task learning methods. Existing HFML methods also disregard the impact of gradient conflicts on multi-task optimization during the federated aggregation process. In this work, we propose an innovative framework called FedBone, which enables the construction of large-scale models with better generalization from the perspective of server-client split learning and gradient projection. We split the entire model into two components: a large-scale general model (referred to as the general model) on the cloud server and multiple task-specific models (referred to as the client model) on edge clients, solving the problem of insufficient computing power on edge clients. The conflicting gradient projection technique is used to enhance the generalization of the large-scale general model between different tasks. The proposed framework is evaluated on two benchmark datasets and a real ophthalmic dataset. Comprehensive results demonstrate that FedBone efficiently adapts to heterogeneous local tasks of each client and outperforms existing federated learning algorithms in most dense prediction and classification tasks with off-the-shelf computational resources on the client side.