Reading Order Independent Metrics for Information Extraction in Handwritten Documents

TL;DR

This paper introduces new reading order independent metrics for evaluating information extraction in handwritten documents, addressing biases caused by reading order errors and providing a more application-relevant assessment.

Contribution

It proposes and publicly releases a set of metrics that are independent of reading order, improving evaluation fairness for handwritten document information extraction.

Findings

Metrics are less biased by reading order errors.

Recommended minimal set of metrics for accurate evaluation.

Enhanced evaluation reflecting real-world application needs.

Abstract

Information Extraction processes in handwritten documents tend to rely on obtaining an automatic transcription and performing Named Entity Recognition (NER) over such transcription. For this reason, in publicly available datasets, the performance of the systems is usually evaluated with metrics particular to each dataset. Moreover, most of the metrics employed are sensitive to reading order errors. Therefore, they do not reflect the expected final application of the system and introduce biases in more complex documents. In this paper, we propose and publicly release a set of reading order independent metrics tailored to Information Extraction evaluation in handwritten documents. In our experimentation, we perform an in-depth analysis of the behavior of the metrics to recommend what we consider to be the minimal set of metrics to evaluate a task correctly.