DTELS: Towards Dynamic Granularity of Timeline Summarization

TL;DR

This paper introduces DTELS, a new approach for creating adaptive news timelines with variable granularity, supported by a comprehensive benchmark, a large dataset, and experiments showing LLM-based solutions are promising but still imperfect.

Contribution

The paper proposes the DTELS paradigm for dynamic timeline summarization, along with a new benchmark, dataset, and evaluation framework to advance research in adaptive news summarization.

Findings

LLM-based solutions are effective for DTELS tasks.

Current LLMs struggle with maintaining consistency and informativeness.

The benchmark facilitates future research in dynamic timeline summarization.

Abstract

The rapid proliferation of online news has posed significant challenges in tracking the continuous development of news topics. Traditional timeline summarization constructs a chronological summary of the events but often lacks the flexibility to meet the diverse granularity needs. To overcome this limitation, we introduce a new paradigm, Dynamic-granularity TimELine Summarization, (DTELS), which aims to construct adaptive timelines based on user instructions or requirements. This paper establishes a comprehensive benchmark for DTLES that includes: (1) an evaluation framework grounded in journalistic standards to assess the timeline quality across four dimensions: Informativeness, Granular Consistency, Factuality, and Coherence; (2) a large-scale, multi-source dataset with multiple granularity timeline annotations based on a consensus process to facilitate authority; (3) extensive experiments and analysis with two proposed solutions based on Large Language Models (LLMs) and existing state-of-the-art TLS methods. The experimental results demonstrate the effectiveness of LLM-based solutions. However, even the most advanced LLMs struggle to consistently generate timelines that are both informative and granularly consistent, highlighting the challenges of the DTELS task.