CrisisBench: Benchmarking Crisis-related Social Media Datasets for Humanitarian Information Processing

TL;DR

CrisisBench consolidates multiple crisis-related social media datasets and provides benchmarks to facilitate the development of better models for humanitarian crisis informatics tasks.

Contribution

This work merges existing datasets and offers standardized benchmarks, enabling more effective comparison and advancement in crisis informatics research.

Findings

Benchmark datasets improve model comparison

Deep learning architectures evaluated on crisis data

Consolidated datasets support better model training

Abstract

Time-critical analysis of social media streams is important for humanitarian organizations for planing rapid response during disasters. The \textit{crisis informatics} research community has developed several techniques and systems for processing and classifying big crisis-related data posted on social media. However, due to the dispersed nature of the datasets used in the literature (e.g., for training models), it is not possible to compare the results and measure the progress made towards building better models for crisis informatics tasks. In this work, we attempt to bridge this gap by combining various existing crisis-related datasets. We consolidate eight human-annotated datasets and provide 166.1k and 141.5k tweets for \textit{informativeness} and \textit{humanitarian} classification tasks, respectively. We believe that the consolidated dataset will help train more sophisticated models. Moreover, we provide benchmarks for both binary and multiclass classification tasks using several deep learning architecrures including, CNN, fastText, and transformers. We make the dataset and scripts available at: https://crisisnlp.qcri.org/crisis_datasets_benchmarks.html