GLIMPSE: Do Large Vision-Language Models Truly Think With Videos or Just Glimpse at Them?

TL;DR

GLIMPSE is a new benchmark designed to evaluate whether large vision-language models can genuinely understand and reason with videos, revealing that current models still largely rely on superficial frame analysis.

Contribution

The paper introduces GLIMPSE, a comprehensive video understanding benchmark with carefully crafted questions that require full video reasoning, highlighting the gap in current LVLMs' video comprehension capabilities.

Findings

Human accuracy on GLIMPSE is 94.82%.

Current LVLMs, like GPT-3, achieve only 66.43%.

Models struggle to move beyond superficial frame analysis.

Abstract

Existing video benchmarks often resemble image-based benchmarks, with question types like "What actions does the person perform throughout the video?" or "What color is the woman's dress in the video?" For these, models can often answer by scanning just a few key frames, without deep temporal reasoning. This limits our ability to assess whether large vision-language models (LVLMs) can truly think with videos rather than perform superficial frame-level analysis. To address this, we introduce GLIMPSE, a benchmark specifically designed to evaluate whether LVLMs can genuinely think with videos. Unlike prior benchmarks, GLIMPSE emphasizes comprehensive video understanding beyond static image cues. It consists of 3,269 videos and over 4,342 highly visual-centric questions across 11 categories, including Trajectory Analysis, Temporal Reasoning, and Forensics Detection. All questions are carefully crafted by human annotators and require watching the entire video and reasoning over full video context-this is what we mean by thinking with video. These questions cannot be answered by scanning selected frames or relying on text alone. In human evaluations, GLIMPSE achieves 94.82% accuracy, but current LVLMs face significant challenges. Even the best-performing model, GPT-o3, reaches only 66.43%, highlighting that LVLMs still struggle to move beyond surface-level reasoning to truly think with videos.