ChroKnowledge: Unveiling Chronological Knowledge of Language Models in Multiple Domains

TL;DR

This paper introduces ChroKnowBench, a benchmark dataset and evaluation framework for assessing the chronological knowledge of language models across multiple domains and temporal aspects, highlighting their strengths and limitations.

Contribution

It presents a novel benchmark and sampling-based framework to evaluate LLMs' ability to recall and adapt to evolving and static knowledge over time.

Findings

LLMs' temporal knowledge recall varies with training data format.

Models partially recall knowledge and often cut off at temporal boundaries.

The proposed prompting method improves knowledge elicitation across different LLMs.

Abstract

Large language models (LLMs) have brought significant changes to many aspects of our lives. However, assessing and ensuring their chronological knowledge remains challenging. Existing approaches fall short in addressing the temporal adaptability of knowledge, often relying on a fixed time-point view. To overcome this, we introduce ChroKnowBench, a benchmark dataset designed to evaluate chronologically accumulated knowledge across three key aspects: multiple domains, time dependency, temporal state. Our benchmark distinguishes between knowledge that evolves (e.g., personal history, scientific discoveries, amended laws) and knowledge that remain constant (e.g., mathematical truths, commonsense facts). Building on this benchmark, we present ChroKnowledge (Chronological Categorization of Knowledge), a novel sampling-based framework for evaluating LLMs' non-parametric chronological knowledge. Our evaluation led to the following observations: (1) The ability of eliciting temporal knowledge varies depending on the data format that model was trained on. (2) LLMs partially recall knowledge or show a cut-off at temporal boundaries rather than recalling all aspects of knowledge correctly. Thus, we apply our ChroKnowPrompt, an in-depth prompting to elicit chronological knowledge by traversing step-by-step through the surrounding time spans. We observe that it successfully recalls objects across both open-source and proprietary LLMs, demonstrating versatility, though it faces challenges with dynamic datasets and unstructured formats.