FinCriticalED: A Visual Benchmark for Financial Fact-Level OCR

TL;DR

FinCriticalED is a new visual benchmark designed to evaluate whether OCR and vision-language models accurately preserve critical financial evidence beyond lexical similarity, highlighting gaps in current systems' reliability.

Contribution

The paper introduces FinCriticalED, a fact-centric benchmark with expert-annotated financial facts, and develops a protocol for assessing model fidelity in financial OCR tasks.

Findings

Numerical values and monetary units are most vulnerable to errors.

Critical errors are concentrated in visually complex, mixed-layout documents.

There is a significant gap between lexical accuracy and factual reliability in current models.

Abstract

Recent progress in multimodal large language models (MLLMs) has substantially improved document understanding, yet strong optical character recognition (OCR) performance on surface metrics does not guarantee faithful preservation of decision-critical evidence. This limitation is especially consequential in financial documents, where small visual errors can induce discrete shifts in meaning. To study this gap, we introduce FinCriticalED (Financial Critical Error Detection), a fact-centric visual benchmark for evaluating whether OCR and vision-language systems preserve financially critical evidence beyond lexical similarity. FinCriticalED contains 859 real-world financial document pages with 9,481 expert-annotated facts spanning five critical field types: numeric, temporal, monetary unit, reporting entity, and financial concept. We formulate the task as structured OCR with fact-level verification, and develop a Deterministic-Rule-Guided LLM-as-Judge protocol to assess whether model outputs preserve annotated facts in context. We benchmark 13 systems spanning OCR pipelines, specialized OCR VLMs, open-source MLLMs, and proprietary MLLMs. Results reveal a clear gap between lexical accuracy and factual reliability, with numerical values and monetary units emerging as the most vulnerable fact types, and critical errors concentrating in visually complex, mixed-layout documents with distinct failure patterns across model families. Overall, FinCriticalED provides a rigorous benchmark for trustworthy financial OCR and a practical testbed for evidence fidelity in high-stakes multimodal document understanding. Benchmark and dataset details available at https://the-finai.github.io/FinCriticalED/