CHEW: A Dataset of CHanging Events in Wikipedia

TL;DR

CHEW is a new dataset capturing changing events in Wikipedia text, used to evaluate large language models' ability to understand timelines and detect meaning shifts, revealing their limitations in temporal reasoning.

Contribution

The paper introduces CHEW, a novel dataset for studying temporal changes in Wikipedia, and demonstrates its utility in probing LLMs' timeline understanding and meaning shift detection.

Findings

LLMs struggle with accurate timeline construction despite available temporal info.

CHEW-derived embeddings help identify meaning shifts in text.

CHEW provides a valuable resource for temporal and semantic analysis.

Abstract

We introduce CHEW, a novel dataset of changing events in Wikipedia expressed in naturally occurring text. We use CHEW for probing LLMs for their timeline understanding of Wikipedia entities and events in generative and classification experiments. Our results suggest that LLMs, despite having temporal information available, struggle to construct accurate timelines. We further show the usefulness of CHEW-derived embeddings for identifying meaning shift.