When Good OCR Is Not Enough: Benchmarking OCR Robustness for Retrieval-Augmented Generation

TL;DR

This paper introduces a new OCR benchmark for industrial RAG systems, revealing that high OCR accuracy does not always ensure effective downstream retrieval and generation in complex real-world documents.

Contribution

The authors present a comprehensive OCR benchmark for industrial RAG, highlighting the limitations of character-level metrics and analyzing factors affecting downstream performance.

Findings

High OCR accuracy does not guarantee strong RAG performance.

Structural and semantic errors impact retrieval success despite low WER/CER.

Performance degradation is consistent across various OCR models and pipeline configurations.

Abstract

Industrial Retrieval-Augmented Generation (RAG) systems depend on optical character recognition (OCR) to transform visual documents into text. Existing OCR benchmarks rely on character-level metrics, which inadequately measure downstream RAG effectiveness under real-world conditions. We introduce an OCR benchmark for industrial RAG systems covering 11 challenging document types, including extreme layouts, high-resolution pages, complex or watermarked backgrounds, historical documents with non-standard reading orders, visually decorated text, and documents containing tables and mathematical formulas. Evaluating recent SOTA OCR models under a controlled OCR-first RAG pipeline shows clear performance degradation on realistic industrial documents despite strong conventional benchmark scores. We find that high OCR accuracy does not necessarily translate into strong downstream RAG performance: structural and semantic errors can cause substantial retrieval failures even when WER/CER remains low. Further analysis shows that this mismatch is category-dependent, arises through both retrieval-side and downstream generation-side failures, and remains stable across representative OCR-first pipeline choices. The benchmark is publicly available at https://github.com/Qihoo360/InduOCRBench.