DrVideo: Document Retrieval Based Long Video Understanding

TL;DR

DrVideo introduces a novel approach for understanding long videos by converting them into text documents and iteratively retrieving key information, enabling large language models to perform long-range reasoning effectively.

Contribution

The paper presents DrVideo, a system that transforms long videos into text documents and uses an agent-based iterative process for improved long video understanding with LLMs.

Findings

Outperforms state-of-the-art LLM-based methods on multiple long video benchmarks.

Effectively retrieves and reasons over long video content.

Demonstrates significant accuracy improvements on benchmarks.

Abstract

Most of the existing methods for video understanding primarily focus on videos only lasting tens of seconds, with limited exploration of techniques for handling long videos. The increased number of frames in long videos poses two main challenges: difficulty in locating key information and performing long-range reasoning. Thus, we propose DrVideo, a document-retrieval-based system designed for long video understanding. Our key idea is to convert the long-video understanding problem into a long-document understanding task so as to effectively leverage the power of large language models. Specifically, DrVideo first transforms a long video into a coarse text-based long document to initially retrieve key frames and then updates the documents with the augmented key frame information. It then employs an agent-based iterative loop to continuously search for missing information and augment the document until sufficient question-related information is gathered for making the final predictions in a chain-of-thought manner. Extensive experiments on long video benchmarks confirm the effectiveness of our method. DrVideo significantly outperforms existing LLM-based state-of-the-art methods on EgoSchema benchmark (3 minutes), MovieChat-1K benchmark (10 minutes), and the long split of Video-MME benchmark (average of 44 minutes).