Benchmarking Scientific Understanding and Reasoning for Video Generation using VideoScience-Bench

TL;DR

VideoScience-Bench is a novel benchmark designed to evaluate the scientific reasoning and understanding capabilities of video generation models across physics and chemistry scenarios, emphasizing their ability to generate physically and chemically accurate phenomena.

Contribution

The paper introduces the first benchmark to assess video models' scientific reasoning, combining diverse prompts and expert evaluations to measure understanding beyond mere generation quality.

Findings

Strong correlation between VLM-as-a-Judge and human assessments.

Existing models show limited scientific reasoning in generated videos.

Benchmark covers 14 scientific topics with 200 prompts.

Abstract

The next frontier for video generation lies in developing models capable of zero-shot reasoning, where understanding real-world scientific laws is crucial for accurate physical outcome modeling under diverse conditions. However, existing video benchmarks are physical commonsense-based, offering limited insight into video models' scientific reasoning capability. We introduce VideoScience-Bench, a benchmark designed to evaluate undergraduate-level scientific understanding in video models. Each prompt encodes a composite scientific scenario that requires understanding and reasoning across multiple scientific concepts to generate the correct phenomenon. The benchmark comprises 200 carefully curated prompts spanning 14 topics and 103 concepts in physics and chemistry. We conduct expert-annotated evaluations across seven state-of-the-art video models in T2V and I2V settings along five dimensions: Prompt Consistency, Phenomenon Congruency, Correct Dynamism, Immutability, and Spatio-Temporal Continuity. Using a VLM-as-a-Judge to assess video generations, we observe strong correlation with human assessments. To the best of our knowledge, VideoScience-Bench is the first benchmark to evaluate video models not only as generators but also as reasoners, requiring their generations to demonstrate scientific understanding consistent with expected physical and chemical phenomena. Our data and evaluation code are available at: \href{https://github.com/hao-ai-lab/VideoScience}{github.com/hao-ai-lab/VideoScience}.