UR-Bench: A Benchmark for Multi-Hop Reasoning over Ultra-High-Resolution Images

TL;DR

UR-Bench is a new benchmark designed to evaluate multimodal large language models' reasoning abilities on ultra-high-resolution images, addressing a gap in visual reasoning evaluation for complex visual data.

Contribution

We introduce UR-Bench, a comprehensive ultra-high-resolution image reasoning benchmark, along with an agent-based framework and tools for efficient processing, advancing visual reasoning evaluation.

Findings

State-of-the-art models show limited reasoning on ultra-high-resolution images.

Our framework improves reasoning efficiency and accuracy on ultra-high-resolution data.

UR-Bench enables detailed evaluation of visual reasoning capabilities.

Abstract

Recent multimodal large language models (MLLMs) show strong capabilities in visual-language reasoning, yet their performance on ultra-high-resolution imagery remains largely unexplored. Existing visual question answering (VQA) benchmarks typically rely on medium-resolution data, offering limited visual complexity. To bridge this gap, we introduce Ultra-high-resolution Reasoning Benchmark (UR-Bench), a benchmark designed to evaluate the reasoning capabilities of MLLMs under extreme visual information. UR-Bench comprises two major categories, Humanistic Scenes and Natural Scenes, covering four subsets of ultra-high-resolution images with distinct spatial structures and data sources. Each subset contains images ranging from hundreds of megapixels to gigapixels, accompanied by questions organized into three levels, enabling evaluation of models' reasoning capabilities in ultra-high-resolution scenarios. We further propose an agent-based framework in which a language model performs reasoning by invoking external visual tools. In addition, we introduce Semantic Abstraction and Retrieval tools that enable more efficient processing of ultra-high-resolution images. We evaluate state-of-the-art models using both an end-to-end MLLMs and our agent-based framework, demonstrating the effectiveness of our framework.