Do Multimodal RAG Systems Leak Data? A Comprehensive Evaluation of Membership Inference and Image Caption Retrieval Attacks

TL;DR

This paper investigates privacy risks in multimodal Retrieval-Augmented Generation systems, demonstrating potential data leakage through membership inference and caption retrieval attacks, and emphasizes the need for privacy-preserving solutions.

Contribution

It provides an empirical evaluation of privacy vulnerabilities in mRAG systems, highlighting specific attack methods and their implications for data privacy.

Findings

mRAG systems can leak private visual data and metadata.

Membership inference attacks can identify dataset inclusion.

Caption retrieval attacks can extract sensitive image information.

Abstract

The growing adoption of multimodal Retrieval-Augmented Generation (mRAG) pipelines for vision-centric tasks (e.g., visual QA) introduces important privacy challenges. In particular, while mRAG provides a practical capability to connect private datasets and improve model performance, it risks the leakage of private information from these datasets. In this paper, we perform an empirical study to analyze the privacy risks inherent in the mRAG pipeline observed through standard model prompting. Specifically, we implement a case study that attempts to determine whether a visual asset (e.g., image) is included in the mRAG, and, if present, to leak the metadata (e.g., caption) related to it. Our findings highlight the need for privacy-preserving mechanisms and motivate future research on mRAG privacy. Our code is published online: https://github.com/aliwister/mrag-attack-eval.