FedVLM: Scalable Personalized Vision-Language Models through Federated Learning

TL;DR

FedVLM introduces a federated learning framework with personalized LoRA for scalable, privacy-preserving adaptation of vision-language models to heterogeneous client data, significantly improving local performance.

Contribution

The paper proposes FedVLM, a federated LoRA fine-tuning framework with personalized adaptation, addressing data heterogeneity and privacy in decentralized vision-language model training.

Findings

pLoRA improves client-specific performance by 24.5%.

FedVLM enables scalable, privacy-preserving VLM fine-tuning in federated settings.

Significant performance gains in non-iid data scenarios.

Abstract

Vision-language models (VLMs) demonstrate impressive zero-shot and few-shot learning capabilities, making them essential for several downstream tasks. However, fine-tuning these models at scale remains challenging, particularly in federated environments where data is decentralized and non-iid across clients. Existing parameter-efficient tuning methods like LoRA (Low-Rank Adaptation) reduce computational overhead but struggle with heterogeneous client data, leading to suboptimal generalization. To address these challenges, we propose FedVLM, a federated LoRA fine-tuning framework that enables decentralized adaptation of VLMs while preserving model privacy and reducing reliance on centralized training. To further tackle data heterogeneity, we introduce personalized LoRA (pLoRA), which dynamically adapts LoRA parameters to each client's unique data distribution, significantly improving local adaptation while maintaining global model aggregation. Experiments on the RLAIF-V dataset show that pLoRA improves client-specific performance by 24.5% over standard LoRA, demonstrating superior adaptation in non-iid settings. FedVLM provides a scalable and efficient solution for fine-tuning VLMs in federated settings, advancing personalized adaptation in distributed learning scenarios.