An end-to-end TextSpotter with Explicit Alignment and Attention

TL;DR

This paper introduces an end-to-end framework for text detection and recognition in natural images, utilizing explicit alignment and attention mechanisms to improve accuracy and efficiency.

Contribution

It presents a novel text-alignment layer, a character attention mechanism with explicit supervision, and integrates these with a new RNN branch into a unified, trainable model.

Findings

Achieved state-of-the-art end-to-end recognition results on ICDAR2015.

Significant improvements in F-measure over previous methods.

Model also sets new benchmarks in text detection performance.

Abstract

Text detection and recognition in natural images have long been considered as two separate tasks that are processed sequentially. Training of two tasks in a unified framework is non-trivial due to significant dif- ferences in optimisation difficulties. In this work, we present a conceptually simple yet efficient framework that simultaneously processes the two tasks in one shot. Our main contributions are three-fold: 1) we propose a novel text-alignment layer that allows it to precisely compute convolutional features of a text instance in ar- bitrary orientation, which is the key to boost the per- formance; 2) a character attention mechanism is introduced by using character spatial information as explicit supervision, leading to large improvements in recognition; 3) two technologies, together with a new RNN branch for word recognition, are integrated seamlessly into a single model which is end-to-end trainable. This allows the two tasks to work collaboratively by shar- ing convolutional features, which is critical to identify challenging text instances. Our model achieves impressive results in end-to-end recognition on the ICDAR2015 dataset, significantly advancing most recent results, with improvements of F-measure from (0.54, 0.51, 0.47) to (0.82, 0.77, 0.63), by using a strong, weak and generic lexicon respectively. Thanks to joint training, our method can also serve as a good detec- tor by achieving a new state-of-the-art detection performance on two datasets.