VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoning

TL;DR

VideoThinker introduces a synthetic training approach for agentic Video LLMs, enabling adaptive reasoning and tool use to improve long-form video understanding without requiring pre-existing long-video comprehension.

Contribution

It presents a novel synthetic data generation method for training agentic Video LLMs with dynamic reasoning and tool interaction capabilities.

Findings

Outperforms caption-only and baseline video models on long-video benchmarks.

Uses synthetic tool interaction trajectories to train the model.

Demonstrates effective adaptive temporal exploration and multi-step reasoning.

Abstract

Long-form video understanding remains a fundamental challenge for current Video Large Language Models. Most existing models rely on static reasoning over uniformly sampled frames, which weakens temporal localization and leads to substantial information loss in long videos. Agentic tools such as temporal retrieval, spatial zoom, and temporal zoom offer a natural way to overcome these limitations by enabling adaptive exploration of key moments. However, constructing agentic video understanding data requires models that already possess strong long-form video comprehension, creating a circular dependency. We address this challenge with VideoThinker, an agentic Video Large Language Model trained entirely on synthetic tool interaction trajectories. Our key idea is to convert videos into rich captions and employ a powerful agentic language model to generate multi-step tool use sequences in caption space. These trajectories are subsequently grounded back to video by replacing captions with the corresponding frames, yielding a large-scale interleaved video and tool reasoning dataset without requiring any long-form understanding from the underlying model. Training on this synthetic agentic dataset equips VideoThinker with dynamic reasoning capabilities, adaptive temporal exploration, and multi-step tool use. Remarkably, VideoThinker significantly outperforms both caption-only language model agents and strong video model baselines across long-video benchmarks, demonstrating the effectiveness of tool augmented synthetic data and adaptive retrieval and zoom reasoning for long-form video understanding.