T2VWorldBench: A Benchmark for Evaluating World Knowledge in Text-to-Video Generation

TL;DR

T2VWorldBench is a comprehensive benchmark designed to evaluate the ability of text-to-video models to incorporate world knowledge, revealing significant gaps in current models' understanding of factual and semantic consistency.

Contribution

This paper introduces the first systematic evaluation framework for assessing world knowledge in text-to-video models, covering diverse domains and combining human and automated assessments.

Findings

Most models struggle with understanding world knowledge

Current models often generate semantically inconsistent videos

Benchmark reveals critical gaps in factual accuracy in T2V models

Abstract

Text-to-video (T2V) models have shown remarkable performance in generating visually reasonable scenes, while their capability to leverage world knowledge for ensuring semantic consistency and factual accuracy remains largely understudied. In response to this challenge, we propose T2VWorldBench, the first systematic evaluation framework for evaluating the world knowledge generation abilities of text-to-video models, covering 6 major categories, 60 subcategories, and 1,200 prompts across a wide range of domains, including physics, nature, activity, culture, causality, and object. To address both human preference and scalable evaluation, our benchmark incorporates both human evaluation and automated evaluation using vision-language models (VLMs). We evaluated the 10 most advanced text-to-video models currently available, ranging from open source to commercial models, and found that most models are unable to understand world knowledge and generate truly correct videos. These findings point out a critical gap in the capability of current text-to-video models to leverage world knowledge, providing valuable research opportunities and entry points for constructing models with robust capabilities for commonsense reasoning and factual generation.