Know-Show: Benchmarking Video-Language Models on Spatio-Temporal Grounded Reasoning

TL;DR

Know-Show introduces a comprehensive benchmark for evaluating spatio-temporal grounded reasoning in Video-Language Models, highlighting current gaps and proposing a plug-in method to improve interpretability and reliability.

Contribution

The paper presents a new benchmark, Know-Show, for assessing grounded reasoning in video-language models and proposes GRAM, a training-free augmentation method to enhance reasoning capabilities.

Findings

Existing Video-LMs significantly lag behind human reasoning in grounded understanding.

The GRAM plug-in improves models' ability to reason about actions and their semantics.

Know-Show provides a unified framework for evaluating spatial and temporal grounding in videos.

Abstract

Large Video-Language Models (Video-LMs) have achieved impressive progress in multimodal understanding, yet their reasoning remains weakly grounded in space and time. We present Know-Show, a new benchmark designed to evaluate spatio-temporal grounded reasoning, the ability of a model to reason about actions and their semantics while simultaneously grounding its inferences in visual and temporal evidence. Know-Show unifies reasoning and localization within a single evaluation framework consisting of five complementary scenarios across spatial (person, object, person-object, and hand-object) and temporal dimensions. Built from Charades, Action Genome, and Ego4D with 2.5K high-quality human-authored questions, the benchmark exposes significant gaps between current Video-LMs and human reasoning. To bridge this gap, we propose GRAM, a training-free plug-in that augments Video-LMs with fine-grained grounded reasoning through attention-based video token selection and explicit timestamp encoding. Extensive experiments across open and closed Video-LMs (e.g., Qwen, VideoR1, Gemini, and GPT-4o) reveal that existing models struggle to "show what they know" and vice versa. Know-Show establishes a unified standard for assessing grounded reasoning in video-language understanding and provides insights toward developing interpretable and reliable multimodal reasoning systems. We have released the dataset at https://github.com/LUNAProject22/Know-Show, and the code will be released in the same repository.