Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding

TL;DR

This paper introduces Deep Video Discovery, an agentic search framework utilizing LLMs and adaptive tool use to improve understanding of long-form videos, achieving state-of-the-art results on benchmark datasets.

Contribution

The paper presents a novel agentic search approach that adaptively orchestrates tools for long video understanding, surpassing previous methods in accuracy.

Findings

Achieves 74.2% accuracy on LVBench dataset.

Improves to 76.0% accuracy with transcripts.

Demonstrates superior performance over prior methods.

Abstract

Long-form video understanding presents significant challenges due to extensive temporal-spatial complexity and the difficulty of question answering under such extended contexts. While Large Language Models (LLMs) have demonstrated considerable advancements in video analysis capabilities and long context handling, they continue to exhibit limitations when processing information-dense hour-long videos. To overcome such limitations, we propose the Deep Video Discovery (DVD) agent to leverage an agentic search strategy over segmented video clips. Unlike previous video agents that rely on predefined workflows applied uniformly across different queries, our approach emphasizes the autonomous and adaptive nature of agents. By providing a set of search-centric tools on multi-granular video database, our DVD agent leverages the advanced reasoning capability of LLM to plan on its current observation state, strategically selects tools to orchestrate adaptive workflow for different queries in light of the gathered information. We perform comprehensive evaluation on multiple long video understanding benchmarks that demonstrates our advantage. Our DVD agent achieves state-of-the-art performance on the challenging LVBench dataset, reaching an accuracy of 74.2%, which substantially surpasses all prior works, and further improves to 76.0% with transcripts. The code has been released at https://github.com/microsoft/DeepVideoDiscovery.