A Benchmark for Ultra-High-Resolution Remote Sensing MLLMs

TL;DR

This paper introduces RSHR-Bench, a new super-high-resolution remote sensing benchmark with 5,329 images and diverse tasks, to better evaluate visual understanding and reasoning in high-res scenarios, revealing existing models' limitations.

Contribution

The paper presents RSHR-Bench, a large-scale, high-resolution benchmark for remote sensing visual understanding, addressing limitations of previous benchmarks and enabling more faithful model evaluation.

Findings

Models show performance gaps in super-high-resolution scenarios.

Text-only LLMs can compete with multimodal models on RS reasoning tasks.

Benchmark covers diverse perception and reasoning tasks with rigorous human verification.

Abstract

Multimodal large language models (MLLMs) demonstrate strong perception and reasoning performance on existing remote sensing (RS) benchmarks. However, most prior benchmarks rely on low-resolution imagery, and some high-resolution benchmarks suffer from flawed reasoning-task designs. We show that text-only LLMs can perform competitively with multimodal vision-language models on RS reasoning tasks without access to images, revealing a critical mismatch between current benchmarks and the intended evaluation of visual understanding. To enable faithful assessment, we introduce RSHR-Bench, a super-high-resolution benchmark for RS visual understanding and reasoning. RSHR-Bench contains 5,329 full-scene images with a long side of at least 4,000 pixels, with up to about 3 x 10^8 pixels per image, sourced from widely used RS corpora and UAV collections. We design four task families: multiple-choice VQA, open-ended VQA, image captioning, and single-image evaluation. These tasks cover nine perception categories and four reasoning types, supporting multi-turn and multi-image dialog. To reduce reliance on language priors, we apply adversarial filtering with strong LLMs followed by rigorous human verification. Overall, we construct 3,864 VQA tasks, 3,913 image captioning tasks, and 500 fully human-written or verified single-image evaluation VQA pairs. Evaluations across open-source, closed-source, and RS-specific VLMs reveal persistent performance gaps in super-high-resolution scenarios. Code: https://github.com/Yunkaidang/RSHR