Localize, Group, and Select: Boosting Text-VQA by Scene Text Modeling

TL;DR

This paper introduces LOGOS, a novel model for Text-VQA that improves scene text understanding by localizing, grouping, and selecting relevant text, leading to state-of-the-art results without extra OCR annotations.

Contribution

LOGOS integrates localization, grouping, and selection mechanisms to enhance scene text comprehension in Text-VQA, advancing multimodal reasoning capabilities.

Findings

Outperforms previous state-of-the-art on two Text-VQA benchmarks

Effectively localizes key scene text information

Successfully bridges different modalities for better understanding

Abstract

As an important task in multimodal context understanding, Text-VQA (Visual Question Answering) aims at question answering through reading text information in images. It differentiates from the original VQA task as Text-VQA requires large amounts of scene-text relationship understanding, in addition to the cross-modal grounding capability. In this paper, we propose Localize, Group, and Select (LOGOS), a novel model which attempts to tackle this problem from multiple aspects. LOGOS leverages two grounding tasks to better localize the key information of the image, utilizes scene text clustering to group individual OCR tokens, and learns to select the best answer from different sources of OCR (Optical Character Recognition) texts. Experiments show that LOGOS outperforms previous state-of-the-art methods on two Text-VQA benchmarks without using additional OCR annotation data. Ablation studies and analysis demonstrate the capability of LOGOS to bridge different modalities and better understand scene text.