"What is the value of {templates}?" Rethinking Document Information Extraction Datasets for LLMs

TL;DR

This paper introduces K2Q, a diverse dataset with complex templates for key information extraction, demonstrating that varied question formats improve the robustness of large language models in document understanding tasks.

Contribution

The work presents K2Q, a novel dataset with diverse, intricate templates for KIE, and empirically shows that question diversity enhances model performance and robustness.

Findings

Diverse templates improve model robustness

Training on complex templates outperforms simple ones

Models benefit from varied question formats

Abstract

The rise of large language models (LLMs) for visually rich document understanding (VRDU) has kindled a need for prompt-response, document-based datasets. As annotating new datasets from scratch is labor-intensive, the existing literature has generated prompt-response datasets from available resources using simple templates. For the case of key information extraction (KIE), one of the most common VRDU tasks, past work has typically employed the template "What is the value for the {key}?". However, given the variety of questions encountered in the wild, simple and uniform templates are insufficient for creating robust models in research and industrial contexts. In this work, we present K2Q, a diverse collection of five datasets converted from KIE to a prompt-response format using a plethora of bespoke templates. The questions in K2Q can span multiple entities and be extractive or boolean. We empirically compare the performance of seven baseline generative models on K2Q with zero-shot prompting. We further compare three of these models when training on K2Q versus training on simpler templates to motivate the need of our work. We find that creating diverse and intricate KIE questions enhances the performance and robustness of VRDU models. We hope this work encourages future studies on data quality for generative model training.