VTAgent: Agentic Keyframe Anchoring for Evidence-Aware Video TextVQA

TL;DR

This paper introduces VTAgent, a question-guided framework that explicitly anchors keyframes to improve evidence localization and significantly enhance performance on Video TextVQA benchmarks.

Contribution

The paper proposes a novel agentic keyframe anchoring method that outperforms existing approaches and establishes new state-of-the-art results in Video TextVQA.

Findings

Frame-wise question answering outperforms direct video inference.

Explicit keyframe anchoring improves accuracy by +12.12 on average.

The approach is effective with training-free, supervised fine-tuning, and reinforcement learning.

Abstract

Video text-based visual question answering (Video TextVQA) aims to answer questions by reasoning over visual textual content appearing in videos. Despite the strong multimodal video understanding capabilities of recent Video-LLMs, their performance on existing Video TextVQA benchmarks remains limited. To better understand this gap, we conduct an upper-bound analysis through frame-wise question answering, counting a sample as correct if any frame yields the right answer, which significantly outperforms direct video-based inference and reveals a substantial performance gap. The results suggest that the primary bottleneck lies in the localization of key question-relevant evidence, rather than in reasoning capacity itself. Building on this insight, we propose a question-guided agent framework that explicitly anchors the relevant keyframes before answering. The approach operates effectively in a training-free setting and consistently surpasses direct video inference. With additional supervised fine-tuning (SFT) and reinforcement learning (RL), it achieves an average improvement of +12.12 in accuracy and +11.15 in ANLS across benchmarks, establishing new state-of-the-art results. Our study underscores the critical role of explicit keyframe anchoring for advancing Video TextVQA. The code will be publicly released.