Transformer-Based Approach for Joint Handwriting and Named Entity Recognition in Historical documents

TL;DR

This paper introduces a novel end-to-end transformer-based method for jointly recognizing handwriting and extracting named entities in historical documents, operating at the paragraph level to improve accuracy and context utilization.

Contribution

It presents the first transformer-based approach for handwritten named entity recognition, achieving state-of-the-art results without relying on dictionaries or language models.

Findings

Achieved new state-of-the-art in ICDAR 2017 competition

Operating at paragraph level reduces early segmentation errors

Two-stage training improves prediction accuracy

Abstract

The extraction of relevant information carried out by named entities in handwriting documents is still a challenging task. Unlike traditional information extraction approaches that usually face text transcription and named entity recognition as separate subsequent tasks, we propose in this paper an end-to-end transformer-based approach to jointly perform these two tasks. The proposed approach operates at the paragraph level, which brings two main benefits. First, it allows the model to avoid unrecoverable early errors due to line segmentation. Second, it allows the model to exploit larger bi-dimensional context information to identify the semantic categories, reaching a higher final prediction accuracy. We also explore different training scenarios to show their effect on the performance and we demonstrate that a two-stage learning strategy can make the model reach a higher final prediction accuracy. As far as we know, this work presents the first approach that adopts the transformer networks for named entity recognition in handwritten documents. We achieve the new state-of-the-art performance in the ICDAR 2017 Information Extraction competition using the Esposalles database, for the complete task, even though the proposed technique does not use any dictionaries, language modeling, or post-processing.