Empty Shelves or Lost Keys? Recall Is the Bottleneck for Parametric Factuality

TL;DR

This paper introduces a new framework and benchmark for analyzing factual errors in large language models, revealing that access to encoded knowledge, rather than encoding itself, is the main bottleneck, and that reasoning can improve recall.

Contribution

It proposes a fact-level profiling framework and WikiProfile benchmark to distinguish between knowledge encoding and access issues in LLMs, highlighting the importance of retrieval and reasoning.

Findings

Encoding of facts is nearly saturated in frontier models.

Recall failures are systematic and affect long-tail and reverse questions.

Thinking methods can recover a significant portion of recall failures.

Abstract

Standard factuality evaluations of LLMs treat all errors alike, obscuring whether failures arise from missing knowledge (empty shelves) or from limited access to encoded facts (lost keys). We propose a behavioral framework that profiles factual knowledge at the level of facts rather than questions, characterizing each fact by whether it is encoded, and then by how accessible it is: cannot be recalled, can be directly recalled, or can only be recalled with inference-time computation (thinking). To support such profiling, we introduce WikiProfile, a new benchmark constructed via an automated pipeline with a prompted LLM grounded in web search. Across 4 million responses from 13 LLMs, we find that encoding is nearly saturated in frontier models on our benchmark, with GPT-5 and Gemini-3 encoding 95--98% of facts. However, recall remains a major bottleneck: many errors previously attributed to missing knowledge instead stem from failures to access it. These failures are systematic and disproportionately affect long-tail facts and reverse questions. Finally, we show that thinking improves recall and can recover a substantial fraction of failures, indicating that future gains may rely less on scaling and more on methods that improve how models utilize what they already encode.