Notes on Applicability of GPT-4 to Document Understanding

TL;DR

This paper evaluates GPT-4 models on document understanding tasks, highlighting their strengths with multimodal inputs and identifying limitations such as performance drops on lengthy documents and potential model contamination.

Contribution

It provides a reproducible benchmark of GPT-4 models for document understanding, emphasizing the importance of multimodal inputs and analyzing model limitations.

Findings

GPT-4 Vision Turbo performs well with OCR and images.

Text-only GPT-4 models face challenges in document comprehension.

Performance drops significantly on lengthy documents.

Abstract

We perform a missing, reproducible evaluation of all publicly available GPT-4 family models concerning the Document Understanding field, where it is frequently required to comprehend text spacial arrangement and visual clues in addition to textual semantics. Benchmark results indicate that though it is hard to achieve satisfactory results with text-only models, GPT-4 Vision Turbo performs well when one provides both text recognized by an external OCR engine and document images on the input. Evaluation is followed by analyses that suggest possible contamination of textual GPT-4 models and indicate the significant performance drop for lengthy documents.