Mirage of Mastery: Memorization Tricks LLMs into Artificially Inflated Self-Knowledge

TL;DR

This paper reveals that large language models often rely on memorized solutions to falsely inflate their self-assessed reasoning abilities, especially in science and medicine, raising concerns about their trustworthiness.

Contribution

It introduces a novel framework to distinguish genuine reasoning from memorization in LLMs and highlights the impact of memorization on self-knowledge and generalization.

Findings

LLMs show over 45% inconsistency in self-assessment when faced with perturbed tasks.

Memorization significantly influences LLMs' confidence, especially in science and medicine domains.

Current architectures and training patterns have flaws that affect models' self-knowledge stability.

Abstract

When artificial intelligence mistakes memorization for intelligence, it creates a dangerous mirage of reasoning. Existing studies treat memorization and self-knowledge deficits in LLMs as separate issues and do not recognize an intertwining link that degrades the trustworthiness of LLM responses. In our study, we utilize a novel framework to ascertain if LLMs genuinely learn reasoning patterns from training data or merely memorize them to assume competence across problems of similar complexity focused on STEM domains. Our analysis shows a noteworthy problem in generalization: LLMs draw confidence from memorized solutions to infer a higher self-knowledge about their reasoning ability, which manifests as an over 45% inconsistency in feasibility assessments when faced with self-validated, logically coherent task perturbations. This effect is most pronounced in science and medicine domains, which tend to have maximal standardized jargon and problems, further confirming our approach. Significant wavering within the self-knowledge of LLMs also shows flaws in current architectures and training patterns, highlighting the need for techniques that ensure a balanced, consistent stance on models' perceptions of their own knowledge for maximum AI explainability and trustworthiness. Our code and results are available publicly at https://github.com/Sahil-R-Kale/mirage_of_mastery