CREPE: Coordinate-Aware End-to-End Document Parser

TL;DR

CREPE is a novel OCR-free, coordinate-aware sequence generation model for visual document understanding that integrates text parsing and spatial coordinate extraction, achieving state-of-the-art results across multiple document understanding tasks.

Contribution

CREPE introduces a unified, OCR-free approach with coordinate decoding and weak supervision, advancing document parsing and understanding capabilities.

Findings

Achieves state-of-the-art performance on document parsing tasks.

Successfully applied to layout analysis and visual question answering.

Reduces error propagation compared to OCR-dependent methods.

Abstract

In this study, we formulate an OCR-free sequence generation model for visual document understanding (VDU). Our model not only parses text from document images but also extracts the spatial coordinates of the text based on the multi-head architecture. Named as Coordinate-aware End-to-end Document Parser (CREPE), our method uniquely integrates these capabilities by introducing a special token for OCR text, and token-triggered coordinate decoding. We also proposed a weakly-supervised framework for cost-efficient training, requiring only parsing annotations without high-cost coordinate annotations. Our experimental evaluations demonstrate CREPE's state-of-the-art performances on document parsing tasks. Beyond that, CREPE's adaptability is further highlighted by its successful usage in other document understanding tasks such as layout analysis, document visual question answering, and so one. CREPE's abilities including OCR and semantic parsing not only mitigate error propagation issues in existing OCR-dependent methods, it also significantly enhance the functionality of sequence generation models, ushering in a new era for document understanding studies.