Target Prompting for Information Extraction with Vision Language Model

TL;DR

This paper introduces Target Prompting, a technique for improving information extraction from document images using vision language models by focusing prompts on specific regions of documents.

Contribution

The paper proposes Target Prompting, a novel method that explicitly targets document regions to enhance the accuracy of vision language models in information extraction tasks.

Findings

Target Prompting improves answer accuracy for specific document regions.

Evaluation shows better performance over generic prompting techniques.

Targeted prompts reduce information gaps in model responses.

Abstract

The recent trend in the Large Vision and Language model has brought a new change in how information extraction systems are built. VLMs have set a new benchmark with their State-of-the-art techniques in understanding documents and building question-answering systems across various industries. They are significantly better at generating text from document images and providing accurate answers to questions. However, there are still some challenges in effectively utilizing these models to build a precise conversational system. General prompting techniques used with large language models are often not suitable for these specially designed vision language models. The output generated by such generic input prompts is ordinary and may contain information gaps when compared with the actual content of the document. To obtain more accurate and specific answers, a well-targeted prompt is required by the vision language model, along with the document image. In this paper, a technique is discussed called Target prompting, which focuses on explicitly targeting parts of document images and generating related answers from those specific regions only. The paper also covers the evaluation of response for each prompting technique using different user queries and input prompts.