ESIR: End-to-end Scene Text Recognition via Iterative Image Rectification

TL;DR

This paper introduces ESIR, an end-to-end scene text recognition system that iteratively rectifies perspective and curvature distortions, significantly improving recognition accuracy on distorted scene text images.

Contribution

The paper proposes a novel rectification network with line-fitting transformation and an iterative pipeline for better scene text recognition, handling perspective and curvature distortions effectively.

Findings

Achieves superior recognition performance on distorted scene text images.

Effectively rectifies perspective and curvature distortions.

Robust to parameter initialization and trained with minimal annotations.

Abstract

Automated recognition of texts in scenes has been a research challenge for years, largely due to the arbitrary variation of text appearances in perspective distortion, text line curvature, text styles and different types of imaging artifacts. The recent deep networks are capable of learning robust representations with respect to imaging artifacts and text style changes, but still face various problems while dealing with scene texts with perspective and curvature distortions. This paper presents an end-to-end trainable scene text recognition system (ESIR) that iteratively removes perspective distortion and text line curvature as driven by better scene text recognition performance. An innovative rectification network is developed which employs a novel line-fitting transformation to estimate the pose of text lines in scenes. In addition, an iterative rectification pipeline is developed where scene text distortions are corrected iteratively towards a fronto-parallel view. The ESIR is also robust to parameter initialization and the training needs only scene text images and word-level annotations as required by most scene text recognition systems. Extensive experiments over a number of public datasets show that the proposed ESIR is capable of rectifying scene text distortions accurately, achieving superior recognition performance for both normal scene text images and those suffering from perspective and curvature distortions.