ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models

TL;DR

ConTextual introduces a new dataset for evaluating context-sensitive, text-rich visual reasoning in multimodal models, revealing significant performance gaps and challenges in understanding complex visual and temporal information.

Contribution

The paper presents ConTextual, a novel dataset for benchmarking multimodal models on context-sensitive reasoning with text-rich images, and provides comprehensive evaluation and analysis of current models' capabilities.

Findings

GPT-4V outperforms other models but lags 30.8% behind human performance.

Models struggle with time-related data and infographics.

Models perform well on abstract visual contexts like memes and quotes.

Abstract

Many real-world tasks require an agent to reason jointly over text and visual objects, (e.g., navigating in public spaces), which we refer to as context-sensitive text-rich visual reasoning. Specifically, these tasks require an understanding of the context in which the text interacts with visual elements within an image. However, there is a lack of existing datasets to benchmark the state-of-the-art multimodal models' capability on context-sensitive text-rich visual reasoning. In this paper, we introduce ConTextual, a novel dataset featuring human-crafted instructions that require context-sensitive reasoning for text-rich images. We conduct experiments to assess the performance of 14 foundation models (GPT-4V, Gemini-Pro-Vision, LLaVA-Next) and establish a human performance baseline. Further, we perform human evaluations of the model responses and observe a significant performance gap of 30.8% between GPT-4V (the current best-performing Large Multimodal Model) and human performance. Our fine-grained analysis reveals that GPT-4V encounters difficulties interpreting time-related data and infographics. However, it demonstrates proficiency in comprehending abstract visual contexts such as memes and quotes. Finally, our qualitative analysis uncovers various factors contributing to poor performance including lack of precise visual perception and hallucinations. Our dataset, code, and leaderboard can be found on the project page https://con-textual.github.io/