BadPromptFL: A Novel Backdoor Threat to Prompt-based Federated Learning in Multimodal Models

TL;DR

This paper introduces BadPromptFL, a backdoor attack on prompt-based federated learning in multimodal models, demonstrating high success rates and stealthiness, thereby exposing security vulnerabilities in privacy-preserving collaborative AI systems.

Contribution

It is the first to reveal backdoor vulnerabilities in prompt-based federated learning for multimodal models, highlighting a new attack vector in this emerging domain.

Findings

Achieves over 90% attack success rate

Maintains high stealth and minimal client participation

Effective across multiple datasets and protocols

Abstract

Prompt-based tuning has emerged as a lightweight alternative to full fine-tuning in large vision-language models, enabling efficient adaptation via learned contextual prompts. This paradigm has recently been extended to federated learning settings (e.g., PromptFL), where clients collaboratively train prompts under data privacy constraints. However, the security implications of prompt-based aggregation in federated multimodal learning remain largely unexplored, leaving a critical attack surface unaddressed. In this paper, we introduce \textbf{BadPromptFL}, the first backdoor attack targeting prompt-based federated learning in multimodal contrastive models. In BadPromptFL, compromised clients jointly optimize local backdoor triggers and prompt embeddings, injecting poisoned prompts into the global aggregation process. These prompts are then propagated to benign clients, enabling universal backdoor activation at inference without modifying model parameters. Leveraging the contextual learning behavior of CLIP-style architectures, BadPromptFL achieves high attack success rates (e.g., \(>90\%\)) with minimal visibility and limited client participation. Extensive experiments across multiple datasets and aggregation protocols validate the effectiveness, stealth, and generalizability of our attack, raising critical concerns about the robustness of prompt-based federated learning in real-world deployments.