From Pretrain to Pain: Adversarial Vulnerability of Video Foundation Models Without Task Knowledge

TL;DR

This paper uncovers a new security vulnerability in Video Foundation Models by proposing TVA, an attack method that exploits temporal dynamics without task or data access, demonstrating significant risks across multiple video tasks.

Contribution

The paper introduces TVA, a novel temporal-aware adversarial attack that does not require task-specific data or surrogate models, highlighting a practical security threat to open-source VFMs.

Findings

TVA effectively attacks downstream models and MLLMs across 24 tasks.

Adversarial vulnerabilities are demonstrated without access to training data or model architecture.

The study reveals a significant security risk in deploying open-source VFMs.

Abstract

Large-scale Video Foundation Models (VFMs) has significantly advanced various video-related tasks, either through task-specific models or Multi-modal Large Language Models (MLLMs). However, the open accessibility of VFMs also introduces critical security risks, as adversaries can exploit full knowledge of the VFMs to launch potent attacks. This paper investigates a novel and practical adversarial threat scenario: attacking downstream models or MLLMs fine-tuned from open-source VFMs, without requiring access to the victim task, training data, model query, and architecture. In contrast to conventional transfer-based attacks that rely on task-aligned surrogate models, we demonstrate that adversarial vulnerabilities can be exploited directly from the VFMs. To this end, we propose the Transferable Video Attack (TVA), a temporal-aware adversarial attack method that leverages the temporal representation dynamics of VFMs to craft effective perturbations. TVA integrates a bidirectional contrastive learning mechanism to maximize the discrepancy between the clean and adversarial features, and introduces a temporal consistency loss that exploits motion cues to enhance the sequential impact of perturbations. TVA avoids the need to train expensive surrogate models or access to domain-specific data, thereby offering a more practical and efficient attack strategy. Extensive experiments across 24 video-related tasks demonstrate the efficacy of TVA against downstream models and MLLMs, revealing a previously underexplored security vulnerability in the deployment of video models.