Large Scale Scene Text Verification with Guided Attention

TL;DR

This paper introduces a novel end-to-end framework for scene text verification that does not require explicit text detection or recognition, enabling effective text presence determination in images with weak supervision.

Contribution

The work presents the first end-to-end scene text verification framework that learns text-image relationships without bounding box annotations, and introduces the Guided Attention model for real-world applications.

Findings

Outperforms state-of-the-art scene text reading solutions on a challenging Street View Business Matching dataset.

Successfully handles weakly labeled data without explicit scene text detection or recognition.

Provides a new perspective for studying scene text problems through a real-world verification task.

Abstract

Many tasks are related to determining if a particular text string exists in an image. In this work, we propose a new framework that learns this task in an end-to-end way. The framework takes an image and a text string as input and then outputs the probability of the text string being present in the image. This is the first end-to-end framework that learns such relationships between text and images in scene text area. The framework does not require explicit scene text detection or recognition and thus no bounding box annotations are needed for it. It is also the first work in scene text area that tackles suh a weakly labeled problem. Based on this framework, we developed a model called Guided Attention. Our designed model achieves much better results than several state-of-the-art scene text reading based solutions for a challenging Street View Business Matching task. The task tries to find correct business names for storefront images and the dataset we collected for it is substantially larger, and more challenging than existing scene text dataset. This new real-world task provides a new perspective for studying scene text related problems. We also demonstrate the uniqueness of our task via a comparison between our problem and a typical Visual Question Answering problem.