Zero-Shot Classification of Crisis Tweets Using Instruction-Finetuned Large Language Models

TL;DR

This paper evaluates the zero-shot classification capabilities of three large language models on crisis tweets, focusing on identifying informativeness and humanitarian categories, revealing performance variability and dataset quality issues.

Contribution

It introduces an assessment of instruction-finetuned large language models for zero-shot crisis tweet classification, highlighting their varying performance and dataset challenges.

Findings

Models perform better on informativeness without extra info.

Providing event context improves humanitarian classification.

Performance varies significantly across datasets.

Abstract

Social media posts are frequently identified as a valuable source of open-source intelligence for disaster response, and pre-LLM NLP techniques have been evaluated on datasets of crisis tweets. We assess three commercial large language models (OpenAI GPT-4o, Gemini 1.5-flash-001 and Anthropic Claude-3-5 Sonnet) capabilities in zero-shot classification of short social media posts. In one prompt, the models are asked to perform two classification tasks: 1) identify if the post is informative in a humanitarian context; and 2) rank and provide probabilities for the post in relation to 16 possible humanitarian classes. The posts being classified are from the consolidated crisis tweet dataset, CrisisBench. Results are evaluated using macro, weighted, and binary F1-scores. The informative classification task, generally performed better without extra information, while for the humanitarian label classification providing the event that occurred during which the tweet was mined, resulted in better performance. Further, we found that the models have significantly varying performance by dataset, which raises questions about dataset quality.