Truth Neurons
Haohang Li, Yupeng Cao, Yangyang Yu, Jordan W. Suchow, Zining Zhu

TL;DR
This paper identifies specific neurons in language models that encode truthfulness, revealing a shared, layer-wise distribution pattern and demonstrating that manipulating these neurons affects model truthfulness across multiple benchmarks.
Contribution
It introduces a method to locate truth neurons in language models and shows their role in encoding truthfulness in a model-agnostic manner.
Findings
Truth neurons exist across various language models.
Suppressing truth neurons reduces model truthfulness.
Truthfulness encoding is layer-wise and dataset-independent.
Abstract
Despite their remarkable success and deployment across diverse workflows, language models sometimes produce untruthful responses. Our limited understanding of how truthfulness is mechanistically encoded within these models jeopardizes their reliability and safety. In this paper, we propose a method for identifying representations of truthfulness at the neuron level. We show that language models contain truth neurons, which encode truthfulness in a subject-agnostic manner. Experiments conducted across models of varying scales validate the existence of truth neurons, confirming that the encoding of truthfulness at the neuron level is a property shared by many language models. The distribution patterns of truth neurons over layers align with prior findings on the geometry of truthfulness. Selectively suppressing the activations of truth neurons found through the TruthfulQA dataset degrades…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMobile Crowdsensing and Crowdsourcing · Ethics and Social Impacts of AI · Explainable Artificial Intelligence (XAI)
MethodsALIGN
