RescueADI: Adaptive Disaster Interpretation in Remote Sensing Images with Autonomous Agents

TL;DR

This paper introduces RescueADI, a new dataset and autonomous agent-based method for comprehensive disaster scene interpretation in remote sensing images, enabling multi-task analysis with higher accuracy than existing approaches.

Contribution

It presents a novel task of adaptive disaster interpretation, a new dataset with extensive annotations, and an autonomous agent approach driven by large language models for complex multi-task analysis.

Findings

Achieves 9% higher accuracy than existing VQA methods.

Demonstrates effective handling of complex interpretation requests.

Provides a publicly available dataset for future research.

Abstract

Current methods for disaster scene interpretation in remote sensing images (RSIs) mostly focus on isolated tasks such as segmentation, detection, or visual question-answering (VQA). However, current interpretation methods often fail at tasks that require the combination of multiple perception methods and specialized tools. To fill this gap, this paper introduces Adaptive Disaster Interpretation (ADI), a novel task designed to solve requests by planning and executing multiple sequentially correlative interpretation tasks to provide a comprehensive analysis of disaster scenes. To facilitate research and application in this area, we present a new dataset named RescueADI, which contains high-resolution RSIs with annotations for three connected aspects: planning, perception, and recognition. The dataset includes 4,044 RSIs, 16,949 semantic masks, 14,483 object bounding boxes, and 13,424 interpretation requests across nine challenging request types. Moreover, we propose a new disaster interpretation method employing autonomous agents driven by large language models (LLMs) for task planning and execution, proving its efficacy in handling complex disaster interpretations. The proposed agent-based method solves various complex interpretation requests such as counting, area calculation, and path-finding without human intervention, which traditional single-task approaches cannot handle effectively. Experimental results on RescueADI demonstrate the feasibility of the proposed task and show that our method achieves an accuracy 9% higher than existing VQA methods, highlighting its advantages over conventional disaster interpretation approaches. The dataset will be publicly available.