Pay Attention to What You Read: Non-recurrent Handwritten Text-Line Recognition

TL;DR

This paper introduces a non-recurrent transformer-based model for handwritten text-line recognition, achieving high accuracy without the limitations of sequential processing and enabling recognition of out-of-vocabulary words.

Contribution

The paper presents a novel transformer-based approach that replaces recurrent neural networks for handwritten text recognition, allowing parallel processing and out-of-vocabulary word recognition.

Findings

Significant accuracy improvements over prior methods

Effective recognition in few-shot learning scenarios

Ability to recognize out-of-vocabulary words

Abstract

The advent of recurrent neural networks for handwriting recognition marked an important milestone reaching impressive recognition accuracies despite the great variability that we observe across different writing styles. Sequential architectures are a perfect fit to model text lines, not only because of the inherent temporal aspect of text, but also to learn probability distributions over sequences of characters and words. However, using such recurrent paradigms comes at a cost at training stage, since their sequential pipelines prevent parallelization. In this work, we introduce a non-recurrent approach to recognize handwritten text by the use of transformer models. We propose a novel method that bypasses any recurrence. By using multi-head self-attention layers both at the visual and textual stages, we are able to tackle character recognition as well as to learn language-related dependencies of the character sequences to be decoded. Our model is unconstrained to any predefined vocabulary, being able to recognize out-of-vocabulary words, i.e. words that do not appear in the training vocabulary. We significantly advance over prior art and demonstrate that satisfactory recognition accuracies are yielded even in few-shot learning scenarios.