SFA: Scan, Focus, and Amplify toward Guidance-aware Answering for Video TextVQA

TL;DR

This paper introduces SFA, a training-free, Video-LLM-based framework that adaptively scans, focuses on, and amplifies key video regions to improve accuracy in Video TextVQA tasks, achieving state-of-the-art results.

Contribution

SFA is the first training-free, guidance-aware Video-LLM framework for Video TextVQA that enhances answer accuracy by focusing on relevant video cues.

Findings

Achieves state-of-the-art results on multiple Video TextVQA datasets.

Outperforms previous methods by a substantial margin.

Demonstrates strong generalizability across datasets.

Abstract

Video text-based visual question answering (Video TextVQA) task aims to answer questions about videos by leveraging the visual text appearing within the videos. This task poses significant challenges, requiring models to accurately perceive and comprehend scene text that varies in scale, orientation, and clarity across frames, while effectively integrating temporal and semantic context to generate precise answers. Moreover, the model must identify question-relevant textual cues and filter out redundant or irrelevant information to ensure answering is guided by the most relevant and informative cues. To address these challenges, we propose SFA, a training-free framework and the first Video-LLM-based method tailored for Video TextVQA, motivated by the human process of answering questions. By adaptively scanning video frames, selectively focusing on key regions, and directly amplifying them, SFA effectively guides the Video-LLM's attention toward essential cues, enabling it to generate more accurate answers. SFA achieves new state-of-the-art results across several public Video TextVQA datasets and surpasses previous methods by a substantial margin, demonstrating its effectiveness and generalizability.