Multi-Query Focused Disaster Summarization via Instruction-Based Prompting

TL;DR

This paper presents a multi-stream disaster summarization method using retrieval, reranking, and instruction-based prompting with LLaMA-13b, demonstrating strong results in extracting key facts from social media and web sources during emergencies.

Contribution

It introduces a novel combination of retrieval, reranking, and instruction-following summarization with open-source LLMs for disaster event summarization.

Findings

Effective retrieval and reranking pipeline using BM25 and MonoT5

Question Answering prompting improves fact extraction accuracy

Open-source LLMs outperform baseline methods but lag behind proprietary systems

Abstract

Automatic summarization of mass-emergency events plays a critical role in disaster management. The second edition of CrisisFACTS aims to advance disaster summarization based on multi-stream fact-finding with a focus on web sources such as Twitter, Reddit, Facebook, and Webnews. Here, participants are asked to develop systems that can extract key facts from several disaster-related events, which ultimately serve as a summary. This paper describes our method to tackle this challenging task. We follow previous work and propose to use a combination of retrieval, reranking, and an embarrassingly simple instruction-following summarization. The two-stage retrieval pipeline relies on BM25 and MonoT5, while the summarizer module is based on the open-source Large Language Model (LLM) LLaMA-13b. For summarization, we explore a Question Answering (QA)-motivated prompting approach and find the evidence useful for extracting query-relevant facts. The automatic metrics and human evaluation show strong results but also highlight the gap between open-source and proprietary systems.