DisasterInsight: A Multimodal Benchmark for Function-Aware and Grounded Disaster Assessment

TL;DR

DisasterInsight is a comprehensive multimodal benchmark for evaluating vision-language models on realistic disaster assessment tasks, emphasizing functional understanding and grounded reasoning in satellite imagery.

Contribution

The paper introduces DisasterInsight, a new benchmark with a restructured dataset and diverse tasks, along with DI-Chat, a domain-adapted baseline model using efficient fine-tuning techniques.

Findings

Significant performance gaps across models in damage understanding and report generation.

DI-Chat outperforms existing models in damage classification and report quality.

Building-function classification remains a challenging task for current models.

Abstract

Timely interpretation of satellite imagery is critical for disaster response, yet existing vision-language benchmarks for remote sensing largely focus on coarse labels and image-level recognition, overlooking the functional understanding and instruction robustness required in real humanitarian workflows. We introduce DisasterInsight, a multimodal benchmark designed to evaluate vision-language models (VLMs) on realistic disaster analysis tasks. DisasterInsight restructures the xBD dataset into approximately 112K building-centered instances and supports instruction-diverse evaluation across multiple tasks, including building-function classification, damage-level and disaster-type classification, counting, and structured report generation aligned with humanitarian assessment guidelines. To establish domain-adapted baselines, we propose DI-Chat, obtained by fine-tuning existing VLM backbones on disaster-specific instruction data using parameter-efficient Low-Rank Adaptation (LoRA). Extensive experiments on state-of-the-art generic and remote-sensing VLMs reveal substantial performance gaps across tasks, particularly in damage understanding and structured report generation. DI-Chat achieves significant improvements on damage-level and disaster-type classification as well as report generation quality, while building-function classification remains challenging for all evaluated models. DisasterInsight provides a unified benchmark for studying grounded multimodal reasoning in disaster imagery.