Using Neighborhood Context to Improve Information Extraction from Visual Documents Captured on Mobile Phones

TL;DR

This paper introduces a Neighborhood-based Information Extraction (NIE) method that leverages local context in visual documents to enhance extraction accuracy, demonstrating superior performance over existing global context techniques and practical on-device applicability.

Contribution

The paper proposes a novel neighborhood-based approach for information extraction from visual documents, improving accuracy and efficiency over prior global context methods.

Findings

NIE outperforms state-of-the-art global context-based IE techniques.

NIE is effective with both small and large models.

On-device implementation demonstrates practical usability.

Abstract

Information Extraction from visual documents enables convenient and intelligent assistance to end users. We present a Neighborhood-based Information Extraction (NIE) approach that uses contextual language models and pays attention to the local neighborhood context in the visual documents to improve information extraction accuracy. We collect two different visual document datasets and show that our approach outperforms the state-of-the-art global context-based IE technique. In fact, NIE outperforms existing approaches in both small and large model sizes. Our on-device implementation of NIE on a mobile platform that generally requires small models showcases NIE's usefulness in practical real-world applications.