Hierarchical Long Video Understanding with Audiovisual Entity Cohesion and Agentic Search

TL;DR

HAVEN is a novel framework that enhances long video understanding by integrating audiovisual entity cohesion and hierarchical indexing, enabling coherent reasoning and entity tracking across extended video content.

Contribution

The paper introduces HAVEN, a unified approach combining entity-level audiovisual representations with hierarchical indexing and agentic search for improved long-video comprehension.

Findings

Achieves 84.1% accuracy on LVBench, setting a new state-of-the-art.

Demonstrates superior temporal coherence and entity consistency.

Excels in reasoning tasks with 80.1% performance.

Abstract

Long video understanding presents significant challenges for vision-language models due to extremely long context windows. Existing solutions relying on naive chunking strategies with retrieval-augmented generation, typically suffer from information fragmentation and a loss of global coherence. We present HAVEN, a unified framework for long-video understanding that enables coherent and comprehensive reasoning by integrating audiovisual entity cohesion and hierarchical video indexing with agentic search. First, we preserve semantic consistency by integrating entity-level representations across visual and auditory streams, while organizing content into a structured hierarchy spanning global summary, scene, segment, and entity levels. Then we employ an agentic search mechanism to enable dynamic retrieval and reasoning across these layers, facilitating coherent narrative reconstruction and fine-grained entity tracking. Extensive experiments demonstrate that our method achieves good temporal coherence, entity consistency, and retrieval efficiency, establishing a new state-of-the-art with an overall accuracy of 84.1% on LVBench. Notably, it achieves outstanding performance in the challenging reasoning category, reaching 80.1%. These results highlight the effectiveness of structured, multimodal reasoning for comprehensive and context-consistent understanding of long-form videos.