VideoTemp-o3: Harmonizing Temporal Grounding and Video Understanding in Agentic Thinking-with-Videos

TL;DR

VideoTemp-o3 introduces a unified framework for long-video understanding that improves localization, efficiency, and accuracy in agentic thinking-with-videos by jointly modeling grounding and question answering.

Contribution

It presents a novel joint modeling approach with a masking mechanism and reinforcement learning rewards, along with a new dataset and benchmark for long video grounded QA.

Findings

Achieves state-of-the-art performance on long video understanding tasks.

Demonstrates improved localization and reduced hallucinations.

Provides a new benchmark for systematic evaluation.

Abstract

In long-video understanding, conventional uniform frame sampling often fails to capture key visual evidence, leading to degraded performance and increased hallucinations. To address this, recent agentic thinking-with-videos paradigms have emerged, adopting a localize-clip-answer pipeline in which the model actively identifies relevant video segments, performs dense sampling within those clips, and then produces answers. However, existing methods remain inefficient, suffer from weak localization, and adhere to rigid workflows. To solve these issues, we propose VideoTemp-o3, a unified agentic thinking-with-videos framework that jointly models video grounding and question answering. VideoTemp-o3 exhibits strong localization capability, supports on-demand clipping, and can refine inaccurate localizations. Specifically, in the supervised fine-tuning stage, we design a unified masking mechanism that encourages exploration while preventing noise. For reinforcement learning, we introduce dedicated rewards to mitigate reward hacking. Besides, from the data perspective, we develop an effective pipeline to construct high-quality long video grounded QA data, along with a corresponding benchmark for systematic evaluation across various video durations. Experimental results demonstrate that our method achieves remarkable performance on both long video understanding and grounding.