Visual Information Extraction in the Wild: Practical Dataset and End-to-end Solution

TL;DR

This paper introduces a large-scale, diverse dataset for visual information extraction in real-world scenarios and proposes an end-to-end framework using contrastive learning to improve extraction accuracy.

Contribution

The paper provides a new, more challenging dataset for VIE and develops a novel end-to-end model that effectively bridges the gap between OCR and information extraction tasks.

Findings

Existing methods perform worse on the new dataset due to increased complexity.

The proposed method achieves consistent performance improvements on both datasets.

The dataset better reflects real-world challenges for VIE applications.

Abstract

Visual information extraction (VIE), which aims to simultaneously perform OCR and information extraction in a unified framework, has drawn increasing attention due to its essential role in various applications like understanding receipts, goods, and traffic signs. However, as existing benchmark datasets for VIE mainly consist of document images without the adequate diversity of layout structures, background disturbs, and entity categories, they cannot fully reveal the challenges of real-world applications. In this paper, we propose a large-scale dataset consisting of camera images for VIE, which contains not only the larger variance of layout, backgrounds, and fonts but also much more types of entities. Besides, we propose a novel framework for end-to-end VIE that combines the stages of OCR and information extraction in an end-to-end learning fashion. Different from the previous end-to-end approaches that directly adopt OCR features as the input of an information extraction module, we propose to use contrastive learning to narrow the semantic gap caused by the difference between the tasks of OCR and information extraction. We evaluate the existing end-to-end methods for VIE on the proposed dataset and observe that the performance of these methods has a distinguishable drop from SROIE (a widely used English dataset) to our proposed dataset due to the larger variance of layout and entities. These results demonstrate our dataset is more practical for promoting advanced VIE algorithms. In addition, experiments demonstrate that the proposed VIE method consistently achieves the obvious performance gains on the proposed and SROIE datasets.