Instruct-ICL: Instruction-Guided In-Context Learning for Post-Disaster Damage Assessment

TL;DR

This paper enhances the reliability of multimodal large language models for post-disaster visual question answering by using instruction-guided in-context learning and chain-of-thought prompting.

Contribution

It introduces a novel prompting paradigm that uses task-specific instructions generated by one model to guide a second model, improving accuracy in disaster assessment tasks.

Findings

Instruction-guided CoT reasoning improves answer accuracy.

Structured prompting strategies outperform zero-shot baselines.

Evaluation on FloodNet dataset confirms effectiveness.

Abstract

Rapid and accurate situational awareness is essential for effective response during natural disasters, where delays in analysis can significantly hinder decision-making. Training task-specific models for post-disaster assessment is often time-consuming and computationally expensive, making such approaches impractical in time-critical scenarios. Consequently, pretrained multimodal large language models (MLLMs) have emerged as a promising alternative for post-disaster visual question answering (VQA), a task that aims to answer structured questions about visual scenes by jointly reasoning over images and text. While these models demonstrate strong multimodal reasoning capabilities, their responses can be sensitive to prompt formulation, which can limit their reliability in real-world disaster assessment scenarios. In this paper, we investigate whether structured reasoning strategies can improve the reliability of pretrained MLLMs for post-disaster VQA. Specifically, we explore multiple prompting paradigms in which one MLLM is used to generate task-specific instructions that serve as Chain-of-Thought (CoT) guidance for a second MLLM. These instructions are incorporated during answer generation with varying degrees of in-context learning (ICL), enabling the model to leverage both explicit reasoning guidance and contextual examples. We conduct our evaluation on the FloodNet dataset and compare these approaches against a zero-shot baseline. Our results demonstrate that integrating instruction-driven CoT reasoning consistently improves answer accuracy.