Membership Inference Attacks Against Video Large Language Models

TL;DR

This paper introduces a black-box membership inference attack on VideoLLMs, revealing privacy vulnerabilities by analyzing generation behavior at different temperatures and video difficulty features.

Contribution

It presents the first attack specifically targeting VideoLLMs, combining temperature perturbation and video-aware features to infer training membership.

Findings

Achieved 0.68 AUC and 0.63 accuracy in membership inference.

Demonstrated VideoLLMs are vulnerable to black-box membership inference.

Highlights the need for privacy risk mitigation in VideoLLMs.

Abstract

Video large language models (VideoLLMs) are increasingly trained or instruction-tuned on large-scale video--text corpora collected from heterogeneous sources, raising an immediate privacy question: can an external auditor determine whether a particular video was used during training? While membership inference attacks (MIAs) have been studied extensively for classifiers and, more recently, for text and image generation models, the VideoLLM setting remains unexplored. This setting is challenging because black-box auditors observe only generated text, whereas the membership signal is entangled with video-specific factors such as motion complexity and temporal span. In this paper, we present a black-box MIA targeting VideoLLMs that couples temperature-perturbed generation with video-aware difficulty features. Our key intuition is that member samples tend to induce sharper, more brittle generation behavior across decoding temperatures, and that this signal should be interpreted jointly with the intrinsic difficulty of the queried video. Concretely, we query the target model at low and high temperatures, measure the semantic drift between the resulting texts. We evaluate the attack against \texttt{LLaVA-Video-7B-Qwen2-Video-Only} and achieve a member inference AUC of 0.68 and accuracy of 0.63. These results demonstrate that Video-LLMs are vulnerable to black-box membership inference attacks, highlighting an urgent need for the community to systematically evaluate and mitigate privacy risks in VideoLLMs.