Gather and Trace: Rethinking Video TextVQA from an Instance-oriented Perspective

TL;DR

This paper introduces GAT, a novel instance-oriented model for Video TextVQA that improves accuracy and speed by focusing on individual text instances and their spatio-temporal relationships within videos.

Contribution

The paper proposes a new instance-oriented framework with modules for gathering and tracing text instances, significantly enhancing Video TextVQA performance and efficiency.

Findings

GAT outperforms existing methods in accuracy by 3.86%.

GAT achieves ten times faster inference speed than large language models.

Extensive experiments validate the effectiveness and generalization of GAT.

Abstract

Video text-based visual question answering (Video TextVQA) aims to answer questions by explicitly reading and reasoning about the text involved in a video. Most works in this field follow a frame-level framework which suffers from redundant text entities and implicit relation modeling, resulting in limitations in both accuracy and efficiency. In this paper, we rethink the Video TextVQA task from an instance-oriented perspective and propose a novel model termed GAT (Gather and Trace). First, to obtain accurate reading result for each video text instance, a context-aggregated instance gathering module is designed to integrate the visual appearance, layout characteristics, and textual contents of the related entities into a unified textual representation. Then, to capture dynamic evolution of text in the video flow, an instance-focused trajectory tracing module is utilized to establish spatio-temporal relationships between instances and infer the final answer. Extensive experiments on several public Video TextVQA datasets validate the effectiveness and generalization of our framework. GAT outperforms existing Video TextVQA methods, video-language pretraining methods, and video large language models in both accuracy and inference speed. Notably, GAT surpasses the previous state-of-the-art Video TextVQA methods by 3.86\% in accuracy and achieves ten times of faster inference speed than video large language models. The source code is available at https://github.com/zhangyan-ucas/GAT.