MIRA: An LLM-Assisted Benchmark for Multi-Category Integrated Retrieval

TL;DR

MIRA is a new benchmark for multi-category integrated retrieval, using real user queries and LLMs to evaluate diverse scholarly data sources in a unified framework.

Contribution

It introduces a large-scale, realistic IR benchmark covering multiple scholarly categories, utilizing LLMs for relevance assessment to reduce costs.

Findings

Built on real user queries for realism

Covers four distinct scholarly categories

Uses LLMs for topic description and relevance assessment

Abstract

Users increasingly expect modern search systems to offer a unified interface that seamlessly retrieves information from diverse data sources and formats. However, current information retrieval (IR) evaluation benchmarks have not kept pace with this development, primarily due to the lack of test collections that represent the diversity of contemporary search domains. We address this critical gap with MIRA, a novel benchmark based on a large-scale social science search platform. MIRA is designed for category-aware ranking across heterogeneous categories - Publications, Research Data, Variables, and Instruments & Tools - within a single, unified evaluation framework. The proposed collection is distinctive in several ways: (1) it is built upon real user queries, providing a more realistic basis for evaluation; (2) it covers scholarly items from four distinct categories, enabling multi-faceted evaluation; and (3) it leverages a Large Language Model to generate topic descriptions and narratives, as well as for relevance assessment with respect to these topics, substantially reducing the labor and cost of test collection generation. We release this resource to benefit the community by providing a foundational testbed for the research on multi-faceted, category-aware, integrated, or cross-category information retrieval.