Joint Recognition of Handwritten Text and Named Entities with a Neural End-to-end Model

TL;DR

This paper introduces a neural end-to-end model that jointly recognizes handwritten text and extracts named entities, reducing error propagation and improving information extraction from historical handwritten documents.

Contribution

It presents a novel unified neural network architecture for simultaneous handwritten text recognition and named entity recognition, eliminating the need for separate modules.

Findings

Comparable performance to state-of-the-art methods without dictionaries or language models

Effective across different encoding strategies and processing levels

Demonstrates robustness on historical marriage records

Abstract

When extracting information from handwritten documents, text transcription and named entity recognition are usually faced as separate subsequent tasks. This has the disadvantage that errors in the first module affect heavily the performance of the second module. In this work we propose to do both tasks jointly, using a single neural network with a common architecture used for plain text recognition. Experimentally, the work has been tested on a collection of historical marriage records. Results of experiments are presented to show the effect on the performance for different configurations: different ways of encoding the information, doing or not transfer learning and processing at text line or multi-line region level. The results are comparable to state of the art reported in the ICDAR 2017 Information Extraction competition, even though the proposed technique does not use any dictionaries, language modeling or post processing.