Reasoning-OCR: Can Large Multimodal Models Solve Complex Logical Reasoning Problems from OCR Cues?

TL;DR

This paper introduces the Reasoning-OCR benchmark to evaluate large multimodal models' ability to perform complex logical reasoning tasks based on OCR cues across diverse visual scenarios, highlighting current limitations and future research directions.

Contribution

The paper presents a new benchmark, Reasoning-OCR, designed to assess LMMs' complex reasoning capabilities from OCR cues, covering six visual scenarios with 150 challenging questions.

Findings

Current LMMs show limited reasoning performance on Reasoning-OCR.

Benchmark reveals gaps in models' ability to handle complex OCR-based reasoning.

Insights suggest need for improved reasoning modules in multimodal models.

Abstract

Large Multimodal Models (LMMs) have become increasingly versatile, accompanied by impressive Optical Character Recognition (OCR) related capabilities. Existing OCR-related benchmarks emphasize evaluating LMMs' abilities of relatively simple visual question answering, visual-text parsing, etc. However, the extent to which LMMs can deal with complex logical reasoning problems based on OCR cues is relatively unexplored. To this end, we introduce the Reasoning-OCR benchmark, which challenges LMMs to solve complex reasoning problems based on the cues that can be extracted from rich visual-text. Reasoning-OCR covers six visual scenarios and encompasses 150 meticulously designed questions categorized into six reasoning challenges. Additionally, Reasoning-OCR minimizes the impact of field-specialized knowledge. Our evaluation offers some insights for proprietary and open-source LMMs in different reasoning challenges, underscoring the urgent to improve the reasoning performance. We hope Reasoning-OCR can inspire and facilitate future research on enhancing complex reasoning ability based on OCR cues. Reasoning-OCR is publicly available at https://github.com/Hxyz-123/ReasoningOCR.