RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World

TL;DR

This paper introduces a benchmark for evaluating how well large language models adapt to continuously changing real-world knowledge, revealing limitations of current methods and proposing a time-aware retrieval approach to improve temporal consistency.

Contribution

The work presents a new benchmark for real-world knowledge evolution and proposes Chronos, a time-aware retrieval method that enhances temporal consistency without extra training.

Findings

Most existing methods struggle with continuous knowledge drift.

Vanilla RAG and learning-based approaches face catastrophic forgetting.

Chronos improves temporal consistency in LLMs without additional training.

Abstract

Large language models (LLMs) acquire most of their knowledge during pretraining, which ties them to a fixed snapshot of the world and makes adaptation to continuously evolving knowledge challenging. As facts, entities, and events change over time, models may experience continuous knowledge drift, resulting not only in outdated predictions but also in temporally inconsistent reasoning. Although existing approaches, such as continual finetuning, knowledge editing, and retrieval-augmented generation (RAG), aim to update or supplement model knowledge, they are rarely evaluated in settings that reflect chronological, evolving, and real-world knowledge evolution. In this work, we introduce a new benchmark of real-world dynamic events, constructed from time-stamped evidence that captures how knowledge evolves over time, which enables systematic evaluation of model adaptation under continuous knowledge drift. The benchmark reveals that most existing methods, including vanilla RAG and several learning-based approaches, struggle under this setting, exposing critical limitations such as catastrophic forgetting and temporal inconsistency. To mitigate these limitations, we propose a time-aware retrieval baseline, Chronos, which progressively organizes retrieved evidence into an Event Evolution Graph to enable more temporally consistent understanding in LLMs without additional training. Overall, this work provides a foundation for analyzing and advancing LLM adaptation to continuous knowledge drift in realistic settings.