Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Data

TL;DR

This paper proposes a federated prompt-tuning framework tailored for multi-modal, heterogeneous, and incomplete client data, effectively aligning and aggregating prompts across diverse data distributions to improve multimodal learning.

Contribution

It introduces a novel federated prompt-tuning approach that handles heterogeneous and incomplete multimodal data, bridging federated learning and multi-modal prompt-tuning.

Findings

Outperforms state-of-the-art baselines on diverse benchmarks.

Effectively aligns prompt instructions across clients and modalities.

Handles missing features in multi-modal federated datasets.

Abstract

This paper introduces a generalized federated prompt-tuning framework for practical scenarios where local datasets are multi-modal and exhibit different distributional patterns of missing features at the input level. The proposed framework bridges the gap between federated learning and multi-modal prompt-tuning which have traditionally focused on either uni-modal or centralized data. A key challenge in this setting arises from the lack of semantic alignment between prompt instructions that encode similar distributional patterns of missing data across different clients. To address this, our framework introduces specialized client-tuning and server-aggregation designs that simultaneously optimize, align, and aggregate prompt-tuning instructions across clients and data modalities. This allows prompt instructions to complement one another and be combined effectively. Extensive evaluations on diverse multimodal benchmark datasets demonstrate that our work consistently outperforms state-of-the-art (SOTA) baselines.