CC-OCR V2: Benchmarking Large Multimodal Models for Literacy in Real-world Document Processing

TL;DR

CC-OCR V2 introduces a comprehensive benchmark for evaluating large multimodal models on real-world OCR tasks, highlighting their current limitations in practical document processing scenarios.

Contribution

The paper presents CC-OCR V2, a new challenging benchmark tailored to real-world OCR applications, with extensive experiments showing current models' performance gaps.

Findings

Current LMMs perform poorly on real-world OCR tasks.

State-of-the-art models show significant performance degradation.

Benchmark includes 7,093 high-difficulty samples across 5 OCR-centric tracks.

Abstract

Large Multimodal Models (LMMs) have recently shown strong performance on Optical Character Recognition (OCR) tasks, demonstrating their promising capability in document literacy. However, their effectiveness in real-world applications remains underexplored, as existing benchmarks adopt task scopes misaligned with practical applications and assume homogeneous acquisition conditions. To address this gap, we introduce CC-OCR V2, a comprehensive and challenging OCR benchmark tailored to real-world document processing. CC-OCR V2 focuses on practical enterprise document processing tasks and incorporates hard and corner cases that are critical yet underrepresented in prior benchmarks, covering 5 major OCR-centric tracks: text recognition, document parsing, document grounding, key information extraction, and document question answering, comprising 7,093 high-difficulty samples. Extensive experiments on 14 advanced LMMs reveal that current models fall short of real-world application requirements. Even state-of-the-art LMMs exhibit substantial performance degradation across diverse tasks and scenarios. These findings reveal a significant gap between performance on current benchmarks and effectiveness in real-world applications. We release the full dataset and evaluation toolkit at https://github.com/eioss/CC-OCR-V2.