DisastIR: A Comprehensive Information Retrieval Benchmark for Disaster Management

TL;DR

DisastIR is a new comprehensive benchmark designed specifically for evaluating information retrieval systems in disaster management, addressing the unique challenges and needs of this domain.

Contribution

It introduces the first large-scale, domain-specific IR benchmark for disaster management, with diverse queries and tasks to better evaluate model performance in this critical area.

Findings

Significant performance variation among state-of-the-art models.

No single model performs best across all disaster-related tasks.

Disaster-specific IR benchmarks reveal gaps in general-domain models.

Abstract

Effective disaster management requires timely access to accurate and contextually relevant information. Existing Information Retrieval (IR) benchmarks, however, focus primarily on general or specialized domains, such as medicine or finance, neglecting the unique linguistic complexity and diverse information needs encountered in disaster management scenarios. To bridge this gap, we introduce DisastIR, the first comprehensive IR evaluation benchmark specifically tailored for disaster management. DisastIR comprises 9,600 diverse user queries and more than 1.3 million labeled query-passage pairs, covering 48 distinct retrieval tasks derived from six search intents and eight general disaster categories that include 301 specific event types. Our evaluations of 30 state-of-the-art retrieval models demonstrate significant performance variances across tasks, with no single model excelling universally. Furthermore, comparative analyses reveal significant performance gaps between general-domain and disaster management-specific tasks, highlighting the necessity of disaster management-specific benchmarks for guiding IR model selection to support effective decision-making in disaster management scenarios. All source codes and DisastIR are available at https://github.com/KaiYin97/Disaster_IR.