How often do Answers Change? Estimating Recency Requirements in Question Answering

TL;DR

This paper introduces RecencyQA, a dataset and taxonomy for understanding how often answers in question answering change, highlighting the challenges LLMs face with non-stationary, time-sensitive questions and the importance of recency modeling.

Contribution

It proposes a recency-stationarity taxonomy and creates RecencyQA, a dataset for analyzing how answer recency affects LLM performance, emphasizing the need for recency-aware QA systems.

Findings

Non-stationary questions are more challenging for LLMs.

Difficulty increases with higher answer update frequency.

Recency and context dependence are crucial for temporal reasoning.

Abstract

Large language models (LLMs) often rely on outdated knowledge when answering time-sensitive questions, leading to confident yet incorrect responses. Without explicit signals indicating whether up-to-date information is required, models struggle to decide when to retrieve external evidence, how to reason about stale facts, and how to rank answers by their validity. Existing benchmarks either periodically refresh answers or rely on fixed templates, but they do not reflect on how frequently answers change or whether a question inherently requires up-to-date information. To address this gap, we introduce a recency-stationarity taxonomy that categorizes questions by how often their answers change and whether this change frequency is time-invariant or context-dependent. Building on this taxonomy, we present RecencyQA, a dataset of 4,031 open-domain questions annotated with recency and stationarity labels. Through human evaluation and empirical analysis, we show that non-stationary questions, i.e., those where context changes the recency requirement, are significantly more challenging for LLMs, with difficulty increasing as update frequency rises. By explicitly modeling recency and context dependence, RecencyQA enables fine-grained benchmarking and analysis of temporal reasoning beyond binary notions of freshness, and provides a foundation for developing recency-aware and context-sensitive question answering systems.