Monitoring Critical Infrastructure Facilities During Disasters Using Large Language Models

TL;DR

This paper investigates using Large Language Models to monitor critical infrastructure facilities during disasters by analyzing social media data, highlighting their strengths and limitations in classification and inference tasks.

Contribution

It demonstrates the application of open-source LLMs for disaster-related CIF monitoring and evaluates their performance in real-world social media data analysis.

Findings

LLMs perform well in classification tasks

Challenges exist in inference tasks with complex prompts

Insights into LLM strengths and weaknesses in disaster scenarios

Abstract

Critical Infrastructure Facilities (CIFs), such as healthcare and transportation facilities, are vital for the functioning of a community, especially during large-scale emergencies. In this paper, we explore a potential application of Large Language Models (LLMs) to monitor the status of CIFs affected by natural disasters through information disseminated in social media networks. To this end, we analyze social media data from two disaster events in two different countries to identify reported impacts to CIFs as well as their impact severity and operational status. We employ state-of-the-art open-source LLMs to perform computational tasks including retrieval, classification, and inference, all in a zero-shot setting. Through extensive experimentation, we report the results of these tasks using standard evaluation metrics and reveal insights into the strengths and weaknesses of LLMs. We note that although LLMs perform well in classification tasks, they encounter challenges with inference tasks, especially when the context/prompt is complex and lengthy. Additionally, we outline various potential directions for future exploration that can be beneficial during the initial adoption phase of LLMs for disaster response tasks.