OCR Hinders RAG: Evaluating the Cascading Impact of OCR on Retrieval-Augmented Generation

TL;DR

This paper introduces OHRBench, a benchmark to evaluate how OCR errors affect the performance of Retrieval-augmented Generation systems, revealing current OCR solutions are inadequate for high-quality knowledge base construction.

Contribution

The paper presents OHRBench, the first benchmark for analyzing OCR's cascading impact on RAG, including a large dataset and systematic evaluation of OCR noise effects.

Findings

Current OCR solutions are insufficient for RAG knowledge base construction.

OCR noise significantly degrades RAG performance.

A clear relationship exists between OCR noise levels and RAG effectiveness.

Abstract

Retrieval-augmented Generation (RAG) enhances Large Language Models (LLMs) by integrating external knowledge to reduce hallucinations and incorporate up-to-date information without retraining. As an essential part of RAG, external knowledge bases are commonly built by extracting structured data from unstructured PDF documents using Optical Character Recognition (OCR). However, given the imperfect prediction of OCR and the inherent non-uniform representation of structured data, knowledge bases inevitably contain various OCR noises. In this paper, we introduce OHRBench, the first benchmark for understanding the cascading impact of OCR on RAG systems. OHRBench includes 8,561 carefully selected unstructured document images from seven real-world RAG application domains, along with 8,498 Q&A pairs derived from multimodal elements in documents, challenging existing OCR solutions used for RAG. To better understand OCR's impact on RAG systems, we identify two primary types of OCR noise: Semantic Noise and Formatting Noise and apply perturbation to generate a set of structured data with varying degrees of each OCR noise. Using OHRBench, we first conduct a comprehensive evaluation of current OCR solutions and reveal that none is competent for constructing high-quality knowledge bases for RAG systems. We then systematically evaluate the impact of these two noise types and demonstrate the trend relationship between the degree of OCR noise and RAG performance. Our OHRBench, including PDF documents, Q&As, and the ground truth structured data are released at: https://github.com/opendatalab/OHR-Bench