Factual Self-Awareness in Language Models: Representation, Robustness, and Scaling

TL;DR

This paper reveals that large language models possess an internal self-awareness mechanism that helps them assess the correctness of factual recall during generation, which emerges during training and is robust to formatting changes.

Contribution

It provides evidence of intrinsic self-monitoring signals in LLMs that influence factual correctness, enhancing interpretability and reliability.

Findings

LLMs encode linear features indicating factual correctness in residual streams.

Self-awareness signals are robust to minor formatting variations.

Self-awareness emerges rapidly during training and peaks in intermediate layers.

Abstract

Factual incorrectness in generated content is one of the primary concerns in ubiquitous deployment of large language models (LLMs). Prior findings suggest LLMs can (sometimes) detect factual incorrectness in their generated content (i.e., fact-checking post-generation). In this work, we provide evidence supporting the presence of LLMs' internal compass that dictate the correctness of factual recall at the time of generation. We demonstrate that for a given subject entity and a relation, LLMs internally encode linear features in the Transformer's residual stream that dictate whether it will be able to recall the correct attribute (that forms a valid entity-relation-attribute triplet). This self-awareness signal is robust to minor formatting variations. We investigate the effects of context perturbation via different example selection strategies. Scaling experiments across model sizes and training dynamics highlight that self-awareness emerges rapidly during training and peaks in intermediate layers. These findings uncover intrinsic self-monitoring capabilities within LLMs, contributing to their interpretability and reliability.