Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptation

TL;DR

This paper introduces PSSFL, a practical semi-supervised federated learning framework that enables effective foundation model adaptation on privacy-sensitive edge devices with limited resources and unlabeled data, using the novel FedMox architecture.

Contribution

It proposes FedMox, a novel federated mixture-of-experts framework with a spatial router and Soft-Mixture strategy for efficient FM adaptation under resource and data constraints.

Findings

FedMox improves object detection performance in autonomous driving datasets.

The approach reduces memory costs on edge devices.

Experimental results demonstrate effective FM adaptation in privacy-sensitive scenarios.

Abstract

Foundation models (FMs) exhibit remarkable generalization but require adaptation to downstream tasks, particularly in privacy-sensitive applications. Due to data privacy regulations, cloud-based FMs cannot directly access private edge data, limiting their adaptation. Federated learning (FL) provides a privacy-aware alternative, but existing FL approaches overlook the constraints imposed by edge devices -- namely, limited computational resources and the scarcity of labeled data. To address these challenges, we introduce Practical Semi-Supervised Federated Learning (PSSFL), where edge devices hold only unlabeled, low-resolution data, while the server has limited labeled, high-resolution data. In this setting, we propose the Federated Mixture of Experts (FedMox), a novel framework that enhances FM adaptation in FL. FedMox tackles computational and resolution mismatch challenges via a sparse Mixture-of-Experts architecture, employing a spatial router to align features across resolutions and a Soft-Mixture strategy to stabilize semi-supervised learning. We take object detection as a case study, and experiments on real-world autonomous driving datasets demonstrate that FedMox effectively adapts FMs under PSSFL, significantly improving performance with constrained memory costs on edge devices. Our work paves the way for scalable and privacy-preserving FM adaptation in federated scenarios.