Are LLMs Really Not Knowledgeable? Mining the Submerged Knowledge in LLMs' Memory

TL;DR

This paper reveals that large language models often retain correct knowledge even when giving incorrect answers, and introduces new metrics and insights to better evaluate and utilize their latent knowledge in question answering tasks.

Contribution

The study uncovers hidden correct knowledge in LLMs despite incorrect outputs, and proposes Hits@k as a new metric to evaluate this latent knowledge independently of answer surface form.

Findings

LLMs often have more factual knowledge than standard QA accuracy suggests.

Prompting strategies allowing 'unsure' outputs can suppress correct answers.

Hits@k effectively measures latent knowledge retention in LLMs.

Abstract

Large language models (LLMs) have shown promise as parametric knowledge bases, but often underperform on question answering (QA) tasks due to hallucinations and uncertainty. While prior work attributes these failures to knowledge gaps in the model's parameters, we uncover a complementary phenomenon: LLMs frequently retain correct knowledge even when generating incorrect or "unsure" answers. By analyzing the token-level output distributions, we find that correct answers often appear among high-probability candidates, despite not being selected. Motivated by this, we propose Hits@k, a novel metric to evaluate latent knowledge retention independent of answer surface form. Our experiments reveal that LLMs possess significantly more factual knowledge than is reflected by standard QA accuracy. Building on these insights, we further examine the prevailing few-shot QA paradigm. We find that prompting strategies which allow "unsure" outputs can inadvertently suppress correct answers by discouraging low-confidence generation. We design a set of quantitative experiments to measure this suppression effect, offering practical guidance for future prompt and decoding design in knowledge-intensive tasks.