Retrieval Augmented Structured Generation: Business Document Information Extraction As Tool Use

TL;DR

This paper introduces Retrieval Augmented Structured Generation (RASG), a novel framework for Business Document Information Extraction that leverages tool use modeling, achieving state-of-the-art results with large language models.

Contribution

The paper presents RASG as a new approach for BDIE, demonstrates its effectiveness with LLMs surpassing multimodal models, and introduces a practical metric and heuristic for line item recognition.

Findings

RASG achieves SOTA results on BDIE benchmarks.

LLMs with RASG outperform current multimodal models.

Proposed GLIRM metric aligns better with real-world BDIE use cases.

Abstract

Business Document Information Extraction (BDIE) is the problem of transforming a blob of unstructured information (raw text, scanned documents, etc.) into a structured format that downstream systems can parse and use. It has two main tasks: Key-Information Extraction (KIE) and Line Items Recognition (LIR). In this paper, we argue that BDIE is best modeled as a Tool Use problem, where the tools are these downstream systems. We then present Retrieval Augmented Structured Generation (RASG), a novel general framework for BDIE that achieves state of the art (SOTA) results on both KIE and LIR tasks on BDIE benchmarks. The contributions of this paper are threefold: (1) We show, with ablation benchmarks, that Large Language Models (LLMs) with RASG are already competitive with or surpasses current SOTA Large Multimodal Models (LMMs) without RASG on BDIE benchmarks. (2) We propose a new metric class for Line Items Recognition, General Line Items Recognition Metric (GLIRM), that is more aligned with practical BDIE use cases compared to existing metrics, such as ANLS*, DocILE, and GriTS. (3) We provide a heuristic algorithm for backcalculating bounding boxes of predicted line items and tables without the need for vision encoders. Finally, we claim that, while LMMs might sometimes offer marginal performance benefits, LLMs + RASG is oftentimes superior given real-world applications and constraints of BDIE.