Robustness of Structured Data Extraction from Perspectively Distorted Documents

TL;DR

This paper investigates how perspectively distorted documents affect data extraction accuracy of multi-modal LLMs, revealing that distortions significantly degrade performance but can be mitigated by simple rotational correction.

Contribution

It introduces a simplified model of perspective distortion using two parameters and evaluates its impact on LLM-based OCR, highlighting the importance of correction methods.

Findings

Structure-recognition accuracy is significantly affected by perspective distortions.

Rotational correction improves data extraction accuracy.

Distortion modeling with two parameters effectively captures real-world distortions.

Abstract

Optical Character Recognition (OCR) for data extraction from documents is essential to intelligent informatics, such as digitizing medical records and recognizing road signs. Multi-modal Large Language Models (LLMs) can solve this task and have shown remarkable performance. Recently, it has been noticed that the accuracy of data extraction by multi-modal LLMs can be affected when in-plane rotations are present in the documents. However, real-world document images are usually not only in-plane rotated but also perspectively distorted. This study investigates the impacts of such perturbations on the data extraction accuracy for the state-of-the-art model, Gemini-1.5-pro. Because perspective distortions have a high degree of freedom, designing experiments in the same manner as single-parametric rotations is difficult. We observed typical distortions of document images and showed that most of them approximately follow an isosceles-trapezoidal transformation, which allows us to evaluate distortions with a small number of parameters. We were able to reduce the number of independent parameters from eight to two, i.e. rotation angle and distortion ratio. Then, specific entities were extracted from synthetically generated sample documents with varying these parameters. As the performance of LLMs, we evaluated not only a character-recognition accuracy but also a structure-recognition accuracy. Whereas the former represents the classical indicators for optical character recognition, the latter is related to the correctness of reading order. In particular, the structure-recognition accuracy was found to be significantly degraded by document distortion. In addition, we found that this accuracy can be improved by a simple rotational correction. This insight will contribute to the practical use of multi-modal LLMs for OCR tasks.