ADSumm: Annotated Ground-truth Summary Datasets for Disaster Tweet Summarization

TL;DR

This paper introduces ADSumm, a new dataset with annotated summaries for eight disaster events, enhancing supervised disaster tweet summarization and improving model performance by 8-28% in ROUGE scores.

Contribution

The paper provides the first detailed annotation procedure for disaster tweet summaries and adds new features like category, relevance, and key-phrase labels to improve summarization quality.

Findings

Annotated datasets improve summarization performance by up to 28%.

Added feature labels enhance coverage and relevance of summaries.

Ground-truth summaries exhibit high coverage, relevance, and diversity.

Abstract

Online social media platforms, such as Twitter, provide valuable information during disaster events. Existing tweet disaster summarization approaches provide a summary of these events to aid government agencies, humanitarian organizations, etc., to ensure effective disaster response. In the literature, there are two types of approaches for disaster summarization, namely, supervised and unsupervised approaches. Although supervised approaches are typically more effective, they necessitate a sizable number of disaster event summaries for testing and training. However, there is a lack of good number of disaster summary datasets for training and evaluation. This motivates us to add more datasets to make supervised learning approaches more efficient. In this paper, we present ADSumm, which adds annotated ground-truth summaries for eight disaster events which consist of both natural and man-made disaster events belonging to seven different countries. Our experimental analysis shows that the newly added datasets improve the performance of the supervised summarization approaches by 8-28% in terms of ROUGE-N F1-score. Moreover, in newly annotated dataset, we have added a category label for each input tweet which helps to ensure good coverage from different categories in summary. Additionally, we have added two other features relevance label and key-phrase, which provide information about the quality of a tweet and explanation about the inclusion of the tweet into summary, respectively. For ground-truth summary creation, we provide the annotation procedure adapted in detail, which has not been described in existing literature. Experimental analysis shows the quality of ground-truth summary is very good with Coverage, Relevance and Diversity.