MMR: Evaluating Reading Ability of Large Multimodal Models

TL;DR

This paper introduces the MMR benchmark, a comprehensive evaluation tool for assessing large multimodal models' complex reasoning and spatial understanding in text-rich images, revealing their current limitations.

Contribution

The paper presents the first human-annotated, multi-task benchmark for text-rich image understanding, highlighting the gaps in existing LMM capabilities.

Findings

Existing LMMs perform poorly on complex reasoning tasks.

The MMR benchmark exposes limitations of state-of-the-art models.

Current models do not fully understand spatial and contextual information.

Abstract

Large multimodal models (LMMs) have demonstrated impressive capabilities in understanding various types of image, including text-rich images. Most existing text-rich image benchmarks are simple extraction-based question answering, and many LMMs now easily achieve high scores. This means that current benchmarks fail to accurately reflect performance of different models, and a natural idea is to build a new benchmark to evaluate their complex reasoning and spatial understanding abilities. In this work, we propose the Multi-Modal Reading (MMR) benchmark in 11 diverse tasks to evaluate LMMs for text-rich image understanding. MMR is the first text-rich image benchmark built on human annotations with the help of language models. By evaluating several state-of-the-art LMMs, including GPT-4o, it reveals the limited capabilities of existing LMMs underscoring the value of our benchmark.