Temporal reasoning for timeline summarisation in social media

TL;DR

This paper introduces a new dataset and a knowledge distillation approach to enhance temporal reasoning in large language models, significantly improving timeline summarisation quality for social media content.

Contribution

It presents NarrativeReason, a novel dataset for temporal reasoning in narratives, and a knowledge distillation framework that improves timeline summarisation in social media contexts.

Findings

Model outperforms baselines on mental health social media summarisation.

Temporal reasoning enhances summarisation accuracy.

Approach generalizes to out-of-domain social media data.

Abstract

This paper explores whether enhancing temporal reasoning capabilities in Large Language Models (LLMs) can improve the quality of timeline summarisation, the task of summarising long texts containing sequences of events, such as social media threads. We first introduce NarrativeReason, a novel dataset focused on temporal relationships among sequential events within narratives, distinguishing it from existing temporal reasoning datasets that primarily address pair-wise event relationships. Our approach then combines temporal reasoning with timeline summarisation through a knowledge distillation framework, where we first fine-tune a teacher model on temporal reasoning tasks and then distill this knowledge into a student model while simultaneously training it for the task of timeline summarisation. Experimental results demonstrate that our model achieves superior performance on out-of-domain mental health-related timeline summarisation tasks, which involve long social media threads with repetitions of events and a mix of emotions, highlighting the importance and generalisability of leveraging temporal reasoning to improve timeline summarisation.