PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning

TL;DR

PRISM is a comprehensive benchmark with over 10,000 instruction-code pairs designed to evaluate spatial-temporal reasoning in programmatic video generation, highlighting gaps in current language models.

Contribution

Introduces PRISM, a large-scale, real-world benchmark with a novel evaluation framework for assessing spatial and temporal accuracy in programmatic video generation.

Findings

Seven mainstream LLMs show a 41% drop from execution success to spatial correctness.

Current models often produce executable code that lacks spatial coherence.

Evaluation should extend beyond code executability to include spatial-temporal accuracy.

Abstract

Programmatic video generation through code offers geometric precision and temporal coherence beyond pixel-level diffusion models, yet rigorously evaluating whether language models can produce spatially correct animated outputs remains an open problem. We introduce PRISM, a large-scale benchmark of 10,372 human-calibrated instruction-code pairs (20 times larger than prior programmatic video generation benchmarks), grounded in real-world knowledge visualization scenarios across English and Chinese and spanning 437 subject categories. We further propose a funnel-style evaluation framework with four complementary metrics: Code-Level Reliability for executability, Spatial Reasoning for layout correctness over full animation sequences, and Prompt-Aware Dynamic Visual Complexity (PADVC) and Temporal Density (TD) for diagnosing dynamic expression and temporal activity. Systematic evaluation of seven mainstream LLMs reveals a striking Execution-Spatial Gap: the average drop from execution success rate to spatial pass rate is approximately 41%, showing that runnable code does not necessarily yield spatially coherent visual output. These findings show that programmatic video generation evaluation should go beyond executability. PRISM provides a principled benchmark for advancing spatially coherent code generation.