PreSTU: Pre-Training for Scene-Text Understanding

TL;DR

PreSTU is a pre-training method designed to improve scene-text understanding in vision-and-language models by incorporating OCR-aware objectives, leading to better recognition and reasoning about embedded text in images.

Contribution

PreSTU introduces OCR-aware pre-training objectives and a transformer-based architecture to enhance scene-text understanding in V&L models, addressing a gap in existing pre-training methods.

Findings

Improves performance on scene-text related V&L benchmarks.

Enhances text recognition and reasoning in images.

Effective across multiple V&L tasks like VQA and captioning.

Abstract

The ability to recognize and reason about text embedded in visual inputs is often lacking in vision-and-language (V&L) models, perhaps because V&L pre-training methods have often failed to include such an ability in their training objective. In this paper, we propose PreSTU, a novel pre-training recipe dedicated to scene-text understanding (STU). PreSTU introduces OCR-aware pre-training objectives that encourage the model to recognize text from an image and connect it to the rest of the image content. We implement PreSTU using a simple transformer-based encoder-decoder architecture, combined with large-scale image-text datasets with scene text obtained from an off-the-shelf OCR system. We empirically demonstrate the effectiveness of this pre-training approach on eight visual question answering and four image captioning benchmarks.