Document Intelligence Metrics for Visually Rich Document Evaluation

TL;DR

This paper introduces DI-Metrics, a Python library for evaluating Visually-Rich Document models using diverse metrics, and demonstrates its application on the CORD dataset to compare state-of-the-art models.

Contribution

The paper presents DI-Metrics, a comprehensive open-source evaluation library for VRD models, incorporating text, geometric, and hierarchical metrics.

Findings

DI-Metrics effectively evaluates VRD models.

Comparison of three SOTA models and one industry model.

Open-source library available on GitHub.

Abstract

The processing of Visually-Rich Documents (VRDs) is highly important in information extraction tasks associated with Document Intelligence. We introduce DI-Metrics, a Python library devoted to VRD model evaluation comprising text-based, geometric-based and hierarchical metrics for information extraction tasks. We apply DI-Metrics to evaluate information extraction performance using publicly available CORD dataset, comparing performance of three SOTA models and one industry model. The open-source library is available on GitHub.