An Effective Data Augmentation Method by Asking Questions about Scene Text Images

TL;DR

This paper introduces a novel data augmentation approach for scene and handwritten text recognition that uses question-answering tasks to improve OCR models' reasoning about text structure, leading to better accuracy.

Contribution

It proposes a VQA-inspired framework that generates natural-language questions about text attributes to enhance OCR training with structured reasoning tasks.

Findings

Significant reduction in CER and WER on WordArt and Esposalles datasets.

Consistent improvements over baseline OCR models.

Enhanced reasoning about character-level text attributes.

Abstract

Scene text recognition (STR) and handwritten text recognition (HTR) face significant challenges in accurately transcribing textual content from images into machine-readable formats. Conventional OCR models often predict transcriptions directly, which limits detailed reasoning about text structure. We propose a VQA-inspired data augmentation framework that strengthens OCR training through structured question-answering tasks. For each image-text pair, we generate natural-language questions probing character-level attributes such as presence, position, and frequency, with answers derived from ground-truth text. These auxiliary tasks encourage finer-grained reasoning, and the OCR model aligns visual features with textual queries to jointly reason over images and questions. Experiments on WordArt and Esposalles datasets show consistent improvements over baseline models, with significant reductions in both CER and WER. Our code is publicly available at https://github.com/xuyaooo/DataAugOCR.