VMDT: Decoding the Trustworthiness of Video Foundation Models

TL;DR

VMDT introduces a comprehensive benchmark platform for evaluating the trustworthiness of video foundation models across multiple dimensions, revealing critical weaknesses and guiding future improvements.

Contribution

This paper presents the first unified platform, VMDT, for systematically assessing trustworthiness in video models across five key dimensions, filling a significant gap in current evaluation tools.

Findings

Open-source T2V models often generate harmful videos and fail to recognize harmful queries.

Unfairness and privacy risks increase with model scale in V2T models.

Safety levels do not correlate with model size, indicating other factors influence safety.

Abstract

As foundation models become more sophisticated, ensuring their trustworthiness becomes increasingly critical; yet, unlike text and image, the video modality still lacks comprehensive trustworthiness benchmarks. We introduce VMDT (Video-Modal DecodingTrust), the first unified platform for evaluating text-to-video (T2V) and video-to-text (V2T) models across five key trustworthiness dimensions: safety, hallucination, fairness, privacy, and adversarial robustness. Through our extensive evaluation of 7 T2V models and 19 V2T models using VMDT, we uncover several significant insights. For instance, all open-source T2V models evaluated fail to recognize harmful queries and often generate harmful videos, while exhibiting higher levels of unfairness compared to image modality models. In V2T models, unfairness and privacy risks rise with scale, whereas hallucination and adversarial robustness improve -- though overall performance remains low. Uniquely, safety shows no correlation with model size, implying that factors other than scale govern current safety levels. Our findings highlight the urgent need for developing more robust and trustworthy video foundation models, and VMDT provides a systematic framework for measuring and tracking progress toward this goal. The code is available at https://sunblaze-ucb.github.io/VMDT-page/.