On Recognizing Texts of Arbitrary Shapes with 2D Self-Attention

TL;DR

This paper introduces SATRN, a novel Transformer-inspired model utilizing 2D self-attention to effectively recognize texts of arbitrary shapes, including curved and rotated texts, significantly improving accuracy on irregular text benchmarks.

Contribution

The paper presents SATRN, a new architecture that leverages 2D self-attention for scene text recognition, enabling recognition of arbitrarily shaped texts with improved performance.

Findings

Outperforms existing models by 5.7 percentage points on irregular text benchmarks.

Effectively recognizes rotated and multi-line texts.

Provides insights into the model's inner mechanisms and applicability.

Abstract

Scene text recognition (STR) is the task of recognizing character sequences in natural scenes. While there have been great advances in STR methods, current methods still fail to recognize texts in arbitrary shapes, such as heavily curved or rotated texts, which are abundant in daily life (e.g. restaurant signs, product labels, company logos, etc). This paper introduces a novel architecture to recognizing texts of arbitrary shapes, named Self-Attention Text Recognition Network (SATRN), which is inspired by the Transformer. SATRN utilizes the self-attention mechanism to describe two-dimensional (2D) spatial dependencies of characters in a scene text image. Exploiting the full-graph propagation of self-attention, SATRN can recognize texts with arbitrary arrangements and large inter-character spacing. As a result, SATRN outperforms existing STR models by a large margin of 5.7 pp on average in "irregular text" benchmarks. We provide empirical analyses that illustrate the inner mechanisms and the extent to which the model is applicable (e.g. rotated and multi-line text). We will open-source the code.