Federated Knowledge Transfer Fine-tuning Large Server Model with Resource-Constrained IoT Clients

TL;DR

This paper introduces KOALA, a federated learning framework enabling resource-constrained IoT clients to collaboratively fine-tune large models via knowledge transfer, reducing local resource requirements while maintaining performance.

Contribution

The paper proposes KOALA, a novel federated knowledge transfer method allowing resource-limited IoT devices to fine-tune large models through small local models and two joint learning modes.

Findings

Achieves comparable training performance to traditional methods.

Significantly reduces local storage and computing resource needs.

Supports both homogeneous and heterogeneous client capacities.

Abstract

The training of large models, involving fine-tuning, faces the scarcity of high-quality data. Compared to the solutions based on centralized data centers, updating large models in the Internet of Things (IoT) faces challenges in coordinating knowledge from distributed clients by using their private and heterogeneous data. To tackle such a challenge, we propose KOALA (Federated Knowledge Transfer Fine-tuning Large Server Model with Resource-Constrained IoT Clients) to impel the training of large models in IoT. Since the resources obtained by IoT clients are limited and restricted, it is infeasible to locally execute large models and also update them in a privacy-preserving manner. Therefore, we leverage federated learning and knowledge distillation to update large models through collaboration with their small models, which can run locally at IoT clients to process their private data separately and enable large-small model knowledge transfer through iterative learning between the server and clients. Moreover, to support clients with similar or different computing capacities, KOALA is designed with two kinds of large-small model joint learning modes, namely to be homogeneous or heterogeneous. Experimental results demonstrate that compared to the conventional approach, our method can not only achieve similar training performance but also significantly reduce the need for local storage and computing power resources.