Think, Then Verify: A Hypothesis-Verification Multi-Agent Framework for Long Video Understanding

TL;DR

This paper introduces VideoHV-Agent, a hypothesis-verification framework for long video question answering that improves accuracy, interpretability, and efficiency by emphasizing deliberate task formulation before evidence retrieval.

Contribution

The paper proposes a novel structured hypothesis-verification approach for long video understanding, emphasizing reasoning before retrieval to reduce errors and improve interpretability.

Findings

Achieves state-of-the-art accuracy on three benchmarks.

Enhances interpretability and logical soundness.

Reduces computational cost compared to previous methods.

Abstract

Long video understanding is challenging due to dense visual redundancy, long-range temporal dependencies, and the tendency of chain-of-thought and retrieval-based agents to accumulate semantic drift and correlation-driven errors. We argue that long-video reasoning should begin not with reactive retrieval, but with deliberate task formulation: the model must first articulate what must be true in the video for each candidate answer to hold. This thinking-before-finding principle motivates VideoHV-Agent, a framework that reformulates video question answering as a structured hypothesis-verification process. Based on video summaries, a Thinker rewrites answer candidates into testable hypotheses, a Judge derives a discriminative clue specifying what evidence must be checked, a Verifier grounds and tests the clue using localized, fine-grained video content, and an Answer agent integrates validated evidence to produce the final answer. Experiments on three long-video understanding benchmarks show that VideoHV-Agent achieves state-of-the-art accuracy while providing enhanced interpretability, improved logical soundness, and lower computational cost. We make our code publicly available at: https://github.com/Haorane/VideoHV-Agent.