Privacy-preserving Prompt Personalization in Federated Learning for Multimodal Large Language Models

TL;DR

This paper introduces SecFPP, a privacy-preserving protocol for federated prompt personalization in multimodal language models, balancing personalization accuracy with strong privacy guarantees amidst data heterogeneity.

Contribution

SecFPP is a novel secure federated prompt personalization method using hierarchical adaptation and secret sharing, improving privacy without sacrificing performance.

Findings

SecFPP achieves state-of-the-art accuracy under data heterogeneity.

SecFPP outperforms existing privacy-preserving baselines.

SecFPP offers a superior privacy-performance trade-off.

Abstract

Prompt learning is a crucial technique for adapting pre-trained multimodal language models (MLLMs) to user tasks. Federated prompt personalization (FPP) is further developed to address data heterogeneity and local overfitting, however, it exposes personalized prompts - valuable intellectual assets - to privacy risks like prompt stealing or membership inference attacks. Widely-adopted techniques like differential privacy add noise to prompts, whereas degrading personalization performance. We propose SecFPP, a secure FPP protocol harmonizing generalization, personalization, and privacy guarantees. SecFPP employs hierarchical prompt adaptation with domain-level and class-level components to handle multi-granular data imbalance. For privacy, it uses a novel secret-sharing-based adaptive clustering algorithm for domain-level adaptation while keeping class-level components private. While theoretically and empirically secure, SecFPP achieves state-of-the-art accuracy under severe heterogeneity in data distribution. Extensive experiments show it significantly outperforms both non-private and privacy-preserving baselines, offering a superior privacy-performance trade-off.