MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understanding

TL;DR

The paper introduces MRD, a novel, training-free framework that improves high-resolution image understanding in multimodal large language models by combining multi-resolution semantic fusion and open-vocabulary object detection.

Contribution

MRD is the first framework to integrate multi-resolution semantic fusion with open-vocabulary detection for HR image understanding without additional training.

Findings

Achieves state-of-the-art results on HR image benchmarks.

Effectively reduces object fragmentation and semantic bias.

Enhances both single-object and multi-object understanding.

Abstract

Understanding high-resolution (HR) images remains a critical challenge for multimodal large language models (MLLMs). Recent approaches leverage vision-based retrieval-augmented generation (RAG) to retrieve query-relevant crops from HR images, improving understanding capacity of MLLMs. However, this paradigm often leads to object fragmentation, resulting in semantic bias and incomplete retrieval, while also introducing false positives from irrelevant background patches. To address these issues, we propose Multi-resolution Retrieval-Detection (MRD), a training-free framework that enhances HR image understanding from both local and global perspectives. Locally, MRD enforces cross-scale semantic consistency via multi-resolution semantic fusion to mitigate single-resolution bias and alleviate object fragmentation. Globally, it integrates open-vocabulary object detection (OVD) as localization priors within a unified framework. Extensive experiments across multiple MLLMs on HR image benchmarks demonstrate that MRD achieves state-of-the-art (SOTA) performance on both single-object and multi-object understanding tasks. Code will be available at: https://github.com/yf0412/MRD.