LLMpedia: A Transparent Framework to Materialize an LLM's Encyclopedic Knowledge at Scale
Muhammed Saeed, Simon Razniewski

TL;DR
LLMpedia demonstrates that large language models can generate a vast, open, and verifiable encyclopedic knowledge base entirely from their parameters, revealing gaps in current factuality benchmarks.
Contribution
This work introduces LLMpedia, the first fully open parametric encyclopedia, showcasing a scalable framework for materializing and evaluating LLMs' encyclopedic knowledge without retrieval.
Findings
LLMpedia produces approximately 1 million articles from LLMs without retrieval.
Verifiable true rate on Wikipedia subjects is 74.7%, lower than benchmark scores.
LLMpedia achieves higher factuality with less textual similarity compared to Wikipedia.
Abstract
Benchmarks such as MMLU suggest flagship language models approach factuality saturation, with scores above 90\%. We show this picture is incomplete. \emph{LLMpedia} generates encyclopedic articles entirely from parametric memory, producing 1M articles across three model families without retrieval. For gpt-5-mini, the verifiable true rate on Wikipedia-covered subjects is only 74.7\% -- more than 15 percentage points below the benchmark-based picture, consistent with the availability bias of fixed-question evaluation. Beyond Wikipedia, frontier subjects verifiable only through curated web evidence fall further to 63.2\% true rate. Wikipedia covers just 61\% of surfaced subjects, and three model families overlap by only 7.3\% in subject choice. In a capture-trap benchmark inspired by prior analysis of Grokipedia, LLMpedia achieves substantially higher factuality at roughly half the…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Wikis in Education and Collaboration
