Active Learning for Identifying Disaster-Related Tweets: A Comparison with Keyword Filtering and Generic Fine-Tuning

TL;DR

This paper explores the use of Active Learning with RoBERTa models to efficiently identify disaster-related tweets, outperforming keyword filtering and generic fine-tuning, with minimal labeling effort needed.

Contribution

It demonstrates that combining Active Learning with fine-tuned RoBERTa models significantly improves disaster tweet classification performance with less labeled data.

Findings

Active Learning with RoBERTa outperforms keyword filtering.

Few rounds of AL achieve high classification accuracy.

Broadly applicable disaster tweet classifier developed with minimal labeling.

Abstract

Information from social media can provide essential information for emergency response during natural disasters in near real-time. However, it is difficult to identify the disaster-related posts among the large amounts of unstructured data available. Previous methods often use keyword filtering, topic modelling or classification-based techniques to identify such posts. Active Learning (AL) presents a promising sub-field of Machine Learning (ML) that has not been used much in the field of text classification of social media content. This study therefore investigates the potential of AL for identifying disaster-related Tweets. We compare a keyword filtering approach, a RoBERTa model fine-tuned with generic data from CrisisLex, a base RoBERTa model trained with AL and a fine-tuned RoBERTa model trained with AL regarding classification performance. For testing, data from CrisisLex and manually labelled data from the 2021 flood in Germany and the 2023 Chile forest fires were considered. The results show that generic fine-tuning combined with 10 rounds of AL outperformed all other approaches. Consequently, a broadly applicable model for the identification of disaster-related Tweets could be trained with very little labelling effort. The model can be applied to use cases beyond this study and provides a useful tool for further research in social media analysis.