Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layouts

TL;DR

This paper introduces two new datasets, Kleister NDA and Kleister Charity, for key information extraction from long, complex documents, and evaluates state-of-the-art models showing room for improvement.

Contribution

The paper provides the first well-defined datasets for KIE involving long, complex documents with layout features, and benchmarks existing models on these challenging datasets.

Findings

Best model achieved 81.77% F1-score on NDA dataset

Best model achieved 83.57% F1-score on Charity dataset

Datasets challenge current state-of-the-art models

Abstract

The relevance of the Key Information Extraction (KIE) task is increasingly important in natural language processing problems. But there are still only a few well-defined problems that serve as benchmarks for solutions in this area. To bridge this gap, we introduce two new datasets (Kleister NDA and Kleister Charity). They involve a mix of scanned and born-digital long formal English-language documents. In these datasets, an NLP system is expected to find or infer various types of entities by employing both textual and structural layout features. The Kleister Charity dataset consists of 2,788 annual financial reports of charity organizations, with 61,643 unique pages and 21,612 entities to extract. The Kleister NDA dataset has 540 Non-disclosure Agreements, with 3,229 unique pages and 2,160 entities to extract. We provide several state-of-the-art baseline systems from the KIE domain (Flair, BERT, RoBERTa, LayoutLM, LAMBERT), which show that our datasets pose a strong challenge to existing models. The best model achieved an 81.77% and an 83.57% F1-score on respectively the Kleister NDA and the Kleister Charity datasets. We share the datasets to encourage progress on more in-depth and complex information extraction tasks.