DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenes

TL;DR

DisasterVQA is a new benchmark dataset with real-world disaster images and expert-curated questions designed to evaluate and improve vision-language models for disaster response tasks.

Contribution

It introduces DisasterVQA, a comprehensive dataset for perception and reasoning in crisis scenarios, highlighting current model limitations and guiding future development.

Findings

Models perform well on binary questions but struggle with detailed reasoning.

Performance varies across disaster types, regions, and question formats.

DisasterVQA serves as a challenging benchmark for operational disaster response models.

Abstract

Social media imagery provides a low-latency source of situational information during natural and human-induced disasters, enabling rapid damage assessment and response. While Visual Question Answering (VQA) has shown strong performance in general-purpose domains, its suitability for the complex and safety-critical reasoning required in disaster response remains unclear. We introduce DisasterVQA, a benchmark dataset designed for perception and reasoning in crisis contexts. DisasterVQA consists of 1,395 real-world images and 4,405 expert-curated question-answer pairs spanning diverse events such as floods, wildfires, and earthquakes. Grounded in humanitarian frameworks including FEMA ESF and OCHA MIRA, the dataset includes binary, multiple-choice, and open-ended questions covering situational awareness and operational decision-making tasks. We benchmark seven state-of-the-art vision-language models and find performance variability across question types, disaster categories, regions, and humanitarian tasks. Although models achieve high accuracy on binary questions, they struggle with fine-grained quantitative reasoning, object counting, and context-sensitive interpretation, particularly for underrepresented disaster scenarios. DisasterVQA provides a challenging and practical benchmark to guide the development of more robust and operationally meaningful vision-language models for disaster response. The dataset is publicly available at https://doi.org/10.5281/zenodo.18267769.