MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrieval

TL;DR

MM-BRIGHT is a new multimodal benchmark designed to evaluate reasoning-intensive retrieval across diverse technical domains, revealing current models' significant limitations and highlighting the need for advanced visual reasoning capabilities.

Contribution

This paper introduces MM-BRIGHT, the first comprehensive multimodal retrieval benchmark with real-world queries, multiple tasks, and diverse domains, to challenge and advance retrieval models.

Findings

State-of-the-art models perform poorly on MM-BRIGHT tasks.

BM25 achieves only 8.5 nDCG@10 on text-only retrieval.

Multimodal models like Nomic-Vision underperform compared to text-only models.

Abstract

Existing retrieval benchmarks primarily consist of text-based queries where keyword or semantic matching is usually sufficient. Many real-world queries contain multimodal elements, particularly, images such as diagrams, charts, and screenshots that require intensive reasoning to identify relevant documents. To address this gap, we introduce MM-BRIGHT, the first multimodal benchmark for reasoning-intensive retrieval. Our dataset consists of 2,803 real-world queries spanning 29 diverse technical domains, with four tasks of increasing complexity: text-to-text, multimodal-to-text, multimodal-to-image, and multimodal-to-multimodal retrieval. Extensive evaluation reveals that state-of-the-art models struggle across all tasks: BM25 achieves only 8.5 nDCG@10 on text-only retrieval, while the best multimodal model Nomic-Vision reaches just 27.6 nDCG@10 on multimodal-to-text retrieval actually underperforming the best text-only model (DiVeR: 32.2). These results highlight substantial headroom and position MM-BRIGHT as a testbed for next-generation retrieval models that better integrate visual reasoning. Our code and data are available at https://github.com/mm-bright/MM-BRIGHT. See also our official website: https://mm-bright.github.io/.