See then Tell: Enhancing Key Information Extraction with Vision Grounding

TL;DR

This paper introduces STNet, an end-to-end model that enhances key information extraction from visually rich documents by integrating vision grounding with text answers, significantly improving accuracy over traditional OCR-based methods.

Contribution

The paper presents STNet, a novel model utilizing a <see> token for vision grounding in key information extraction, and introduces the TVG dataset created with GPT-4 for training and evaluation.

Findings

Achieves state-of-the-art results on CORD, SROIE, and DocVQA datasets.

Demonstrates improved accuracy in key information extraction tasks.

Provides a new dataset with vision grounding annotations for table QA.

Abstract

In the digital era, the ability to understand visually rich documents that integrate text, complex layouts, and imagery is critical. Traditional Key Information Extraction (KIE) methods primarily rely on Optical Character Recognition (OCR), which often introduces significant latency, computational overhead, and errors. Current advanced image-to-text approaches, which bypass OCR, typically yield plain text outputs without corresponding vision grounding. In this paper, we introduce STNet (See then Tell Net), a novel end-to-end model designed to deliver precise answers with relevant vision grounding. Distinctively, STNet utilizes a unique <see> token to observe pertinent image areas, aided by a decoder that interprets physical coordinates linked to this token. Positioned at the outset of the answer text, the <see> token allows the model to first see-observing the regions of the image related to the input question-and then tell-providing articulated textual responses. To enhance the model's seeing capabilities, we collect extensive structured table recognition datasets. Leveraging the advanced text processing prowess of GPT-4, we develop the TVG (TableQA with Vision Grounding) dataset, which not only provides text-based Question Answering (QA) pairs but also incorporates precise vision grounding for these pairs. Our approach demonstrates substantial advancements in KIE performance, achieving state-of-the-art results on publicly available datasets such as CORD, SROIE, and DocVQA. The code will also be made publicly available.