EEA: Exploration-Exploitation Agent for Long Video Understanding

TL;DR

EEA introduces a semantic-guided hierarchical search framework for long video understanding, balancing exploration and exploitation to improve efficiency and coverage in identifying critical information.

Contribution

The paper presents EEA, a novel agent that dynamically guides exploration in long videos using semantic queries and hierarchical search, enhancing efficiency and accuracy.

Findings

EEA outperforms existing methods on multiple long-video benchmarks.

EEA achieves higher coverage of relevant segments with less computational cost.

EEA demonstrates stable and precise evaluation through adaptive reward combination.

Abstract

Long-form video understanding requires efficient navigation of extensive visual data to pinpoint sparse yet critical information. Current approaches to longform video understanding either suffer from severe computational overhead due to dense preprocessing, or fail to effectively balance exploration and exploitation, resulting in incomplete information coverage and inefficiency. In this work, we introduce EEA, a novel video agent framework that archives exploration-exploitation balance through semantic guidance with hierarchical tree search process. EEA autonomously discovers and dynamically updates task-relevant semantic queries, and collects video frames closely matched to these queries as semantic anchors. During the tree search process, instead of uniform expansion, EEA preferentially explores semantically relevant frames while ensuring sufficient coverage within unknown segments. Moreover, EEA adaptively combines intrinsic rewards from visionlanguage models (VLMs) with semantic priors by explicitly modeling uncertainty to achieve stable and precise evaluation of video segments. Experiments across various long-video benchmarks validate the superior performance and computational efficiency of our proposed method.