Evaluating Robustness of LLMs on Crisis-Related Microblogs across Events, Information Types, and Linguistic Features

TL;DR

This study evaluates the robustness of six large language models in processing disaster-related microblog data, revealing their strengths and weaknesses across different events, information types, and linguistic features, with proprietary models generally outperforming open-source ones.

Contribution

It provides a comprehensive benchmarking of LLMs on crisis-related social media data, analyzing their generalizability, limitations, and the impact of linguistic features across multiple real-world events.

Findings

LLMs like GPT-4 outperform others in generalizability.

Most models struggle with flood-related data and critical information extraction.

Proprietary models outperform open-source models in all tasks.

Abstract

The widespread use of microblogging platforms like X (formerly Twitter) during disasters provides real-time information to governments and response authorities. However, the data from these platforms is often noisy, requiring automated methods to filter relevant information. Traditionally, supervised machine learning models have been used, but they lack generalizability. In contrast, Large Language Models (LLMs) show better capabilities in understanding and processing natural language out of the box. This paper provides a detailed analysis of the performance of six well-known LLMs in processing disaster-related social media data from a large-set of real-world events. Our findings indicate that while LLMs, particularly GPT-4o and GPT-4, offer better generalizability across different disasters and information types, most LLMs face challenges in processing flood-related data, show minimal improvement despite the provision of examples (i.e., shots), and struggle to identify critical information categories like urgent requests and needs. Additionally, we examine how various linguistic features affect model performance and highlight LLMs' vulnerabilities against certain features like typos. Lastly, we provide benchmarking results for all events across both zero- and few-shot settings and observe that proprietary models outperform open-source ones in all tasks.