Question Answering under Temporal Conflict: Evaluating and Organizing Evolving Knowledge with LLMs

TL;DR

This paper evaluates how large language models handle evolving knowledge over time and introduces a framework for organizing temporal information to improve question answering accuracy.

Contribution

It presents new benchmarks for temporal knowledge evaluation and proposes an external memory framework that enhances LLM reasoning with evolving data.

Findings

LLMs struggle with conflicting or outdated facts in temporal contexts.

The proposed memory framework outperforms in-context learning and retrieval-augmented baselines.

Models better handle complex reasoning with structured temporal knowledge.

Abstract

Large language models (LLMs) exhibit remarkable capabilities in question answering and reasoning thanks to their extensive parametric memory. However, their knowledge is inherently limited by the scope of their pre-training data, while real-world information evolves continuously. Updating this knowledge typically requires costly and brittle re-training, or in-context learning (ICL), which becomes impractical at scale given the volume and volatility of modern information. Motivated by these limitations, we investigate how LLMs perform when exposed to temporal text corpora, or documents that reflect evolving knowledge over time, such as sports biographies where facts like a player's "current team" change year by year. To this end, we introduce two new benchmarks: Temporal Wiki, which captures factual drift across historical Wikipedia snapshots, and Unified Clark, which aggregates timestamped news articles to simulate real-world information accumulation. Our analysis reveals that LLMs often struggle to reconcile conflicting or outdated facts and can be misled when multiple versions of a fact appear in context. To address these issues, we propose a lightweight, agentic framework that incrementally builds a structured, external memory from source documents without requiring re-training. This knowledge organization strategy enables models to retrieve and reason over temporally filtered, relevant information at inference time. Empirically, our method outperforms ICL and RAG baselines across both benchmarks, especially on questions requiring more complex reasoning or integration of conflicting facts.