Towards End-to-end Handwritten Document Recognition

TL;DR

This paper introduces an end-to-end approach for handwritten document recognition that processes entire documents without segmentation, achieving state-of-the-art results and handling complex layouts more effectively.

Contribution

It presents the first end-to-end method for recognizing handwritten text and layout in whole documents, moving beyond traditional multi-step segmentation approaches.

Findings

Achieved state-of-the-art results at paragraph level on RIMES 2011, IAM, and READ 2016 datasets.

Outperformed line-level state of the art on the same datasets.

Proposed new metrics for evaluating document-level handwritten recognition.

Abstract

Handwritten text recognition has been widely studied in the last decades for its numerous applications. Nowadays, the state-of-the-art approach consists in a three-step process. The document is segmented into text lines, which are then ordered and recognized. However, this three-step approach has many drawbacks. The three steps are treated independently whereas they are closely related. Errors accumulate from one step to the other. The ordering step is based on heuristic rules which prevent its use for documents with a complex layouts or for heterogeneous documents. The need for additional physical segmentation annotations for training the segmentation stage is inherent to this approach. In this thesis, we propose to tackle these issues by performing the handwritten text recognition of whole document in an end-to-end way. To this aim, we gradually increase the difficulty of the recognition task, moving from isolated lines to paragraphs, and then to whole documents. We proposed an approach at the line level, based on a fully convolutional network, in order to design a first generic feature extraction step for the handwriting recognition task. Based on this preliminary work, we studied two different approaches to recognize handwritten paragraphs. We reached state-of-the-art results at paragraph level on the RIMES 2011, IAM and READ 2016 datasets and outperformed the line-level state of the art on these datasets. We finally proposed the first end-to-end approach dedicated to the recognition of both text and layout, at document level. Characters and layout tokens are sequentially predicted following a learned reading order. We proposed two new metrics we used to evaluate this task on the RIMES 2009 and READ 2016 dataset, at page level and double-page level.