Evaluating Temporal Consistency in Multi-Turn Language Models

TL;DR

This paper introduces ChronoScope, a benchmark for testing how well language models maintain and update temporal context over multiple dialogue turns, revealing significant stability issues.

Contribution

The paper presents a large-scale diagnostic benchmark for evaluating temporal scope stability in multi-turn language models, highlighting prevalent challenges in temporal reasoning.

Findings

Models often drift toward present-day assumptions despite correct knowledge.

Temporal stability issues increase with longer interactions.

Failures persist even with oracle context, indicating a gap in temporal reasoning.

Abstract

Language models are increasingly deployed in interactive settings where users reason about facts over time rather than in isolation. In such scenarios, correct behavior requires models to maintain and update implicit temporal assumptions established earlier in a conversation. We study this challenge through the lens of temporal scope stability: the ability to preserve, override, or transfer time-scoped factual context across dialogue turns. We introduce ChronoScope, a large-scale diagnostic benchmark designed to isolate temporal scope behavior in controlled multi-turn interactions, comprising over one million deterministically generated question chains grounded in Wikidata. ChronoScope evaluates whether models can correctly retain inferred temporal scope when follow-up questions omit explicit time references, spanning implicit carryover, explicit scope switching, cross-entity transfer, and longer temporal trajectories. Through extensive evaluation of state-of-the-art language models, we find that temporal scope stability is frequently violated in controlled multi-turn settings, with models often drifting toward present-day assumptions despite correct underlying knowledge. These failures intensify with interaction length and persist even under oracle context conditions, revealing a gap between single-turn factual accuracy and coherent temporal reasoning under sequential interaction. We make our dataset and evaluation suite publicly available at https://github.com/yashkumaratri/ChronoScope