Beyond the Last Frame: Process-aware Evaluation for Generative Video Reasoning

TL;DR

This paper introduces VIPER, a comprehensive benchmark and a new evaluation metric for generative video reasoning, emphasizing process validation over outcome accuracy to prevent outcome-hacking.

Contribution

It presents VIPER, a multi-task benchmark for process-aware evaluation and POC@r, a hierarchical metric assessing both intermediate reasoning steps and final results.

Findings

State-of-the-art models achieve only about 20% POC@1.0.

Current models exhibit significant outcome-hacking.

There is a large gap between current video generation and true reasoning.

Abstract

Recent breakthroughs in video generation have demonstrated an emerging capability termed Chain-of-Frames (CoF) reasoning, where models resolve complex tasks through the generation of continuous frames. While these models show promise for Generative Video Reasoning (GVR), existing evaluation frameworks often rely on single-frame assessments, which can lead to outcome-hacking, where a model reaches a correct conclusion through an erroneous process. To address this, we propose a process-aware evaluation paradigm. We introduce VIPER, a comprehensive benchmark spanning 16 tasks across temporal, structural, symbolic, spatial, physics, and planning reasoning. Furthermore, we propose Process-outcome Consistency (POC@r), a new metric that utilizes VLM-as-Judge with a hierarchical rubric to evaluate both the validity of the intermediate steps and the final result. Our experiments reveal that state-of-the-art video models achieve POC@1.0 only about 20% and exhibit a significant outcome-hacking. We further explore the impact of test-time scaling and sampling robustness, highlighting a substantial gap between current video generation and true generalized visual reasoning. Our benchmark are released at https://github.com/RUCAIBox/VIPER.