SABRE-FL: Selective and Accurate Backdoor Rejection for Federated Prompt Learning

TL;DR

This paper investigates backdoor vulnerabilities in Federated Prompt Learning for vision-language models, proposing SABRE-FL, a lightweight defense that effectively detects and filters poisoned updates without raw data access.

Contribution

It is the first to analyze backdoor attacks in Federated Prompt Learning and introduces SABRE-FL, a novel embedding-space anomaly detector for robust defense.

Findings

SABRE-FL significantly reduces backdoor success rates across datasets.

The method maintains high accuracy on clean inputs.

It outperforms existing defenses in empirical evaluations.

Abstract

Federated Prompt Learning has emerged as a communication-efficient and privacy-preserving paradigm for adapting large vision-language models like CLIP across decentralized clients. However, the security implications of this setup remain underexplored. In this work, we present the first study of backdoor attacks in Federated Prompt Learning. We show that when malicious clients inject visually imperceptible, learnable noise triggers into input images, the global prompt learner becomes vulnerable to targeted misclassification while still maintaining high accuracy on clean inputs. Motivated by this vulnerability, we propose SABRE-FL, a lightweight, modular defense that filters poisoned prompt updates using an embedding-space anomaly detector trained offline on out-of-distribution data. SABRE-FL requires no access to raw client data or labels and generalizes across diverse datasets. We show, both theoretically and empirically, that malicious clients can be reliably identified and filtered using an embedding-based detector. Across five diverse datasets and four baseline defenses, SABRE-FL outperforms all baselines by significantly reducing backdoor accuracy while preserving clean accuracy, demonstrating strong empirical performance and underscoring the need for robust prompt learning in future federated systems.