InstructOCR: Instruction Boosting Scene Text Spotting

TL;DR

InstructOCR introduces an instruction-based approach to scene text spotting that leverages human language instructions, improving accuracy and flexibility, and achieving state-of-the-art results on key benchmarks.

Contribution

The paper presents a novel instruction-driven framework for scene text spotting that enhances understanding and performance by integrating human language instructions during training and inference.

Findings

Achieves state-of-the-art results on standard benchmarks.

Improves downstream VQA task performance by over 2%.

Demonstrates the effectiveness of instruction strategies in OCR.

Abstract

In the field of scene text spotting, previous OCR methods primarily relied on image encoders and pre-trained text information, but they often overlooked the advantages of incorporating human language instructions. To address this gap, we propose InstructOCR, an innovative instruction-based scene text spotting model that leverages human language instructions to enhance the understanding of text within images. Our framework employs both text and image encoders during training and inference, along with instructions meticulously designed based on text attributes. This approach enables the model to interpret text more accurately and flexibly. Extensive experiments demonstrate the effectiveness of our model and we achieve state-of-the-art results on widely used benchmarks. Furthermore, the proposed framework can be seamlessly applied to scene text VQA tasks. By leveraging instruction strategies during pre-training, the performance on downstream VQA tasks can be significantly improved, with a 2.6% increase on the TextVQA dataset and a 2.1% increase on the ST-VQA dataset. These experimental results provide insights into the benefits of incorporating human language instructions for OCR-related tasks.