Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models

TL;DR

This paper introduces a novel differentially private federated prompt learning method for multimodal large language models, balancing personalization, generalization, and privacy through low-rank factorization and differential privacy techniques.

Contribution

It proposes a DP-FPL approach that uses low-rank factorization and differential privacy to enhance personalization and generalization in multimodal LLMs while preserving privacy.

Findings

Effective privacy-utility tradeoff demonstrated

Outperforms existing benchmarks in experiments

Preserves personalization without sacrificing generalization

Abstract

Multimodal Large Language Models (LLMs) are pivotal in revolutionizing customer support and operations by integrating multiple modalities such as text, images, and audio. Federated Prompt Learning (FPL) is a recently proposed approach that combines pre-trained multimodal LLMs such as vision-language models with federated learning to create personalized, privacy-preserving AI systems. However, balancing the competing goals of personalization, generalization, and privacy remains a significant challenge. Over-personalization can lead to overfitting, reducing generalizability, while stringent privacy measures, such as differential privacy, can hinder both personalization and generalization. In this paper, we propose a Differentially Private Federated Prompt Learning (DP-FPL) approach to tackle this challenge by leveraging a low-rank factorization scheme to capture generalization while maintaining a residual term that preserves expressiveness for personalization. To ensure privacy, we introduce a novel method where we apply local differential privacy to the two low-rank components of the local prompt, and global differential privacy to the global prompt. Our approach mitigates the impact of privacy noise on the model performance while balancing the tradeoff between personalization and generalization. Extensive experiments demonstrate the effectiveness of our approach over other benchmarks.