SceneGATE: Scene-Graph based co-Attention networks for TExt visual question answering

TL;DR

SceneGATE introduces a scene graph-based co-attention network for TextVQA that models semantic relations among objects, OCR tokens, and question words, significantly improving performance on benchmark datasets.

Contribution

The paper presents a novel scene graph-based co-attention network that explicitly models semantic relations for TextVQA, enhancing multimodal interaction understanding.

Findings

Outperforms existing methods on Text-VQA and ST-VQA datasets.

Effectively captures intra- and inter-modal semantic relations.

Improves accuracy by leveraging scene graph and specialized attention modules.

Abstract

Most TextVQA approaches focus on the integration of objects, scene texts and question words by a simple transformer encoder. But this fails to capture the semantic relations between different modalities. The paper proposes a Scene Graph based co-Attention Network (SceneGATE) for TextVQA, which reveals the semantic relations among the objects, Optical Character Recognition (OCR) tokens and the question words. It is achieved by a TextVQA-based scene graph that discovers the underlying semantics of an image. We created a guided-attention module to capture the intra-modal interplay between the language and the vision as a guidance for inter-modal interactions. To make explicit teaching of the relations between the two modalities, we proposed and integrated two attention modules, namely a scene graph-based semantic relation-aware attention and a positional relation-aware attention. We conducted extensive experiments on two benchmark datasets, Text-VQA and ST-VQA. It is shown that our SceneGATE method outperformed existing ones because of the scene graph and its attention modules.