CrisisLTLSum: A Benchmark for Local Crisis Event Timeline Extraction and Summarization

TL;DR

CrisisLTLSum is a new large dataset of local crisis event timelines from social media, designed to benchmark timeline extraction and summarization methods, revealing significant gaps between current models and human performance.

Contribution

The paper introduces CrisisLTLSum, the largest dataset for local crisis event timelines, enabling improved benchmarking of extraction and summarization techniques.

Findings

Significant performance gap between models and humans.

CrisisLTLSum covers four crisis domains.

Dataset and baseline models are publicly available.

Abstract

Social media has increasingly played a key role in emergency response: first responders can use public posts to better react to ongoing crisis events and deploy the necessary resources where they are most needed. Timeline extraction and abstractive summarization are critical technical tasks to leverage large numbers of social media posts about events. Unfortunately, there are few datasets for benchmarking technical approaches for those tasks. This paper presents CrisisLTLSum, the largest dataset of local crisis event timelines available to date. CrisisLTLSum contains 1,000 crisis event timelines across four domains: wildfires, local fires, traffic, and storms. We built CrisisLTLSum using a semi-automated cluster-then-refine approach to collect data from the public Twitter stream. Our initial experiments indicate a significant gap between the performance of strong baselines compared to the human performance on both tasks. Our dataset, code, and models are publicly available.