VCBench: A Streaming Counting Benchmark for Spatial-Temporal State Maintenance in Long Videos

TL;DR

VCBench is a new streaming counting benchmark designed to evaluate and diagnose the ability of video understanding models to maintain spatial-temporal world state, revealing significant deficiencies in current models.

Contribution

It introduces a comprehensive streaming counting benchmark with detailed annotations and metrics for diagnosing spatial-temporal state maintenance in video models.

Findings

Current models struggle with spatial-temporal state maintenance.

Models show deficiencies in periodic event counting.

VCBench provides a diagnostic framework for improvement.

Abstract

Video understanding requires models to continuously track and update world state during playback. While existing benchmarks have advanced video understanding evaluation across multiple dimensions, the observation of how models maintain world state remains insufficient. We propose VCBench, a streaming counting benchmark that repositions counting as a minimal probe for diagnosing world state maintenance capability. We decompose this capability into object counting and event counting, forming 8 fine-grained subcategories. Object counting covers tracking currently visible objects and cumulative unique identities, while event counting covers detecting instantaneous actions and tracking complete activity cycles. VCBench contains 406 videos with frame-by-frame annotations of 10,071 event occurrence moments and object state change moments, generating 1,000 streaming QA pairs with 4,576 query points along timelines. By observing state maintenance trajectories through streaming multi-point queries, we design three complementary metrics to diagnose numerical precision, trajectory consistency, and temporal awareness. Evaluation on mainstream video-language models shows that current models still exhibit significant deficiencies in spatial-temporal state maintenance, particularly struggling with tasks like periodic event counting. VCBench provides a diagnostic framework for measuring and improving state maintenance in video understanding systems. Our code and data are available at https://github.com/buaaplay/VCBench.