FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models

TL;DR

FedVLMBench is a comprehensive benchmark that evaluates federated fine-tuning strategies for vision-language models, addressing privacy concerns and providing insights into architecture and data heterogeneity effects.

Contribution

This work introduces the first systematic benchmark for federated fine-tuning of VLMs, covering multiple architectures, strategies, datasets, and tasks, with extensive experimental analysis.

Findings

A 2-layer MLP connector with concurrent tuning is optimal for encoder-based VLMs in FL.

FL methods are more sensitive to data heterogeneity in vision tasks than in text tasks.

The benchmark offers standardized tools and datasets for advancing privacy-preserving multimodal models.

Abstract

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in cross-modal understanding and generation by integrating visual and textual information. While instruction tuning and parameter-efficient fine-tuning methods have substantially improved the generalization of VLMs, most existing approaches rely on centralized training, posing challenges for deployment in domains with strict privacy requirements like healthcare. Recent efforts have introduced Federated Learning (FL) into VLM fine-tuning to address these privacy concerns, yet comprehensive benchmarks for evaluating federated fine-tuning strategies, model architectures, and task generalization remain lacking. In this work, we present \textbf{FedVLMBench}, the first systematic benchmark for federated fine-tuning of VLMs. FedVLMBench integrates two mainstream VLM architectures (encoder-based and encoder-free), four fine-tuning strategies, five FL algorithms, six multimodal datasets spanning four cross-domain single-task scenarios and two cross-domain multitask settings, covering four distinct downstream task categories. Through extensive experiments, we uncover key insights into the interplay between VLM architectures, fine-tuning strategies, data heterogeneity, and multi-task federated optimization. Notably, we find that a 2-layer multilayer perceptron (MLP) connector with concurrent connector and LLM tuning emerges as the optimal configuration for encoder-based VLMs in FL. Furthermore, current FL methods exhibit significantly higher sensitivity to data heterogeneity in vision-centric tasks than text-centric ones, across both encoder-free and encoder-based VLM architectures. Our benchmark provides essential tools, datasets, and empirical guidance for the research community, offering a standardized platform to advance privacy-preserving, federated training of multimodal foundation models.