VidNum-1.4K: A Comprehensive Benchmark for Video-based Numerical Reasoning

TL;DR

VidNum-1.4K is a new comprehensive benchmark for evaluating video-based numerical reasoning in vision-language models, emphasizing complex multi-step logic in diverse real-world videos.

Contribution

Introduces VidNum-1.4K, a diverse, hierarchical VideoQA benchmark designed to assess genuine numerical reasoning beyond superficial tasks.

Findings

State-of-the-art VLMs perform poorly on VidNum-1.4K, with accuracy mostly below 60%.

Current models lack a stable internal world model for complex numerical reasoning.

VidNum-1.4K serves as a challenging diagnostic for future video numerical reasoning models.

Abstract

Video-based numerical reasoning provides a premier arena for testing whether Vision-Language Models (VLMs) truly "understand" real-world dynamics, as accurate numerical deduction necessitates a profound grasp of temporal events, object permanence, and compositional logic beyond superficial pattern matching. However, existing benchmarks are often confined to narrow domains, such as repetitive athletic motions, or treat simple counting merely as a superficial regression task, failing to assess multi-step numerical logic within the inherent complexity of real-world multimedia content. We introduce VidNum-1.4K, a comprehensive VideoQA benchmark comprising 1,379 strictly human-annotated video-question pairs designed to evaluate genuine numerical reasoning across highly diverse environments, encompassing object, action, and event quantification. The VidNum-1.4K is uniquely structured into a three-level hierarchy that evolves from direct visual perception to video-based compositional numerical reasoning, requiring models to perform arithmetic operations, comparisons, and logical deductions grounded in temporal evidence. Our evaluations across a diverse suite of state-of-the-art VLMs reveal a striking reasoning gap: while the Gemini-3.1-pro barely reaches a 60% accuracy threshold, representative open-source families struggle heavily in the 25%--45% range. These findings demonstrate that current VLMs still lack a stable "internal world model", positioning VidNum-1.4K as a demanding diagnostic testbed for the next generation of numerical video intelligence.