LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge
Heng Zhou, Ao Yu, Yuchen Fan, Jianing Shi, Li Kang, Hejia Geng, Yongting Zhang, Yutao Fan, Yuhao Wu, Tiancheng He, Yiran Qin, Lei Bai, Zhenfei Yin
TL;DR
LiveSearchBench is an automated, scalable benchmark that evaluates large language models on retrieval and reasoning over recent, dynamic knowledge updates, highlighting models' recency gaps.
Contribution
It introduces a fully automated pipeline for creating temporally grounded benchmarks from knowledge updates, enabling ongoing evaluation of models' retrieval capabilities.
Findings
Models perform worse on post-training facts, especially on multi-hop queries.
Retrieval-augmented methods and larger models improve performance but don't fully close the recency gap.
The benchmark emphasizes the importance of up-to-date retrieval and reasoning in LLM evaluation.
Abstract
Evaluating large language models (LLMs) on question answering often relies on static benchmarks that reward memorization and understate the role of retrieval, failing to capture the dynamic nature of world knowledge. We present LiveSearchBench, an automated pipeline for constructing retrieval-dependent benchmarks from recent knowledge updates. Our method computes deltas between successive Wikidata snapshots, filters candidate triples for quality, and synthesizes natural-language questions at three levels of reasoning difficulty, each guaranteed to admit a unique, verifiable answer through SPARQL validation. The pipeline is fully automated, scalable across time, and minimizes human intervention, enabling continual regeneration of temporally grounded benchmarks. Experiments show a pronounced performance drop when models confront facts that post-date pretraining, with the gap most salient…
Peer Reviews
Decision·ICLR 2026 Conference Withdrawn Submission
1. The work addresses a real need for benchmarks that evolve to stay ahead of LLMs' parametric knowledge. 2. The approach of automatic benchmark generation via updated Wikidata snapshots is sensible (but see a minor concern about Wikidata under "weaknesses"). 3. The inclusion of the three question levels is helpful, as it makes it possible to separately evaluate search and reasoning abilities.
1. There is insufficient analysis of the results. Even with retrieval, the best performance on the simplest questions (level 1) is in the low 70s%. I would expect the paper to include analysis of whether the errors come from search errors (this could be done by including an "oracle" search that provides a known correct document) or another source. In the absence of such details, I am wondering how trustworthy the results are. 2. There is insufficient detail provided about the human validatio
[S1] Important task with clear motivation. The paper directly addresses a specific underexplored task in current LLM evaluation: overreliance on static benchmarks that reward memorization. By curating questions in post-dated Wikidata changes, LiveSearchBench offers a principled way to isolate retrieval and reasoning over evolving knowledge. [S2] Methodology is sound and fully automated. The four-stage pipeline (delta extraction -> filtering -> synthesis -> SPARQL verification) is conceptually
[W1] Scale and manual review. Despite claiming full automation, the released set contains only 600 questions with some degree of human verification by five reviewers. This is sufficient for a proof-of-concept but small compared to existing QA benchmarks, potentially limiting statistical reliability and diversity of reasoning types. [W2] Evaluating more models will make the conclusions more convincing. The experiments focus on a small set of model families (Llama3.2, Qwen2.5) and retrieval-based
- The paper is clear and easy to read. - The benchmark construction based on Wikipedia graph updates makes sense to reduce LLM memorization of the knowledge being tested. - Results are well presented, and agree with expectations.
- The novelty of the paper is limited. Dynamic benchmarking is an important direction, but there has been many works such as AntiLeakBench [1], Daily Oracle [2], among others [3, 4]. Among them, one of the most relevant work, AntiLeakBench [1], also works with the Wikipedia updates for constructing knowledge testing samples that are dynamic and not memorized. Compared to the prior research, the proposed work lacks both scale and comprehensiveness in both benchmark construction and evaluation. Pl
* The paper addresses an important problem: once a benchmark is published, models may train on it. An ever-evolving dataset with the latest knowledge is crucial for correctly evaluating RAG systems. * The benchmark creation process is rigorous with solid quality control. Low-quality triples are filtered in the Candidate Filtering step, improving reliability, and the Finalization and Validation step further improves question quality.
* Missing discussion of closely related work. The idea appears very similar to AntiLeakBench [1], especially in methodology: both use Wikipedia as the knowledge base, leverage edit history across snapshots to construct non-contaminated data, and use fully automated QA generation pipelines. The differences between this paper and AntiLeakBench should be clarified. * Limited model families and sizes. It would be better to include closed-source models and more open-source families. * The “fully auto
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Multimodal Machine Learning Applications · Advanced Graph Neural Networks