AVATAAR: Agentic Video Answering via Temporal Adaptive Alignment and Reasoning

TL;DR

AVATAAR is a modular, interpretable framework that enhances long-form video question answering by combining global and local context with iterative reasoning, leading to significant performance improvements.

Contribution

Introduces AVATAAR, a novel framework integrating global/local context, retrieval, and reasoning modules with feedback for improved video understanding.

Findings

Achieves +5.6% in temporal reasoning on CinePile

Gains +8% in theme-based questions

Feedback loop enhances adaptability and performance

Abstract

With the increasing prevalence of video content, effectively understanding and answering questions about long form videos has become essential for numerous applications. Although large vision language models (LVLMs) have enhanced performance, they often face challenges with nuanced queries that demand both a comprehensive understanding and detailed analysis. To overcome these obstacles, we introduce AVATAAR, a modular and interpretable framework that combines global and local video context, along with a Pre Retrieval Thinking Agent and a Rethink Module. AVATAAR creates a persistent global summary and establishes a feedback loop between the Rethink Module and the Pre Retrieval Thinking Agent, allowing the system to refine its retrieval strategies based on partial answers and replicate human-like iterative reasoning. On the CinePile benchmark, AVATAAR demonstrates significant improvements over a baseline, achieving relative gains of +5.6% in temporal reasoning, +5% in technical queries, +8% in theme-based questions, and +8.2% in narrative comprehension. Our experiments confirm that each module contributes positively to the overall performance, with the feedback loop being crucial for adaptability. These findings highlight AVATAAR's effectiveness in enhancing video understanding capabilities. Ultimately, AVATAAR presents a scalable solution for long-form Video Question Answering (QA), merging accuracy, interpretability, and extensibility.