Patchfinder: Leveraging Visual Language Models for Accurate Information Retrieval using Model Uncertainty

TL;DR

PatchFinder is a novel algorithm that enhances information retrieval from noisy scanned documents by leveraging vision language models and a confidence-based patching approach, significantly improving accuracy over existing methods.

Contribution

It introduces a confidence-based patching method for VLMs to improve information extraction from noisy documents, reducing reliance on expensive language models.

Findings

Achieves 94% accuracy on noisy scanned documents.

Outperforms ChatGPT-4o by 18.5 percentage points.

Uses a confidence metric to optimize patch size.

Abstract

For decades, corporations and governments have relied on scanned documents to record vast amounts of information. However, extracting this information is a slow and tedious process due to the sheer volume and complexity of these records. The rise of Vision Language Models (VLMs) presents a way to efficiently and accurately extract the information out of these documents. The current automated workflow often requires a two-step approach involving the extraction of information using optical character recognition software and subsequent usage of large language models for processing this information. Unfortunately, these methods encounter significant challenges when dealing with noisy scanned documents, often requiring computationally expensive language models to handle high information density effectively. In this study, we propose PatchFinder, an algorithm that builds upon VLMs to improve information extraction. First, we devise a confidence-based score, called Patch Confidence, based on the Maximum Softmax Probability of the VLMs' output to measure the model's confidence in its predictions. Using this metric, PatchFinder determines a suitable patch size, partitions the input document into overlapping patches, and generates confidence-based predictions for the target information. Our experimental results show that PatchFinder, leveraging Phi-3v, a 4.2-billion-parameter VLM, achieves an accuracy of 94% on our dataset of 190 noisy scanned documents, outperforming ChatGPT-4o by 18.5 percentage points.