Are the Hidden States Hiding Something? Testing the Limits of Factuality-Encoding Capabilities in LLMs

TL;DR

This paper investigates whether internal states of LLMs encode truthfulness, using more realistic datasets, and finds that generalization remains challenging, highlighting the need for improved factuality evaluation methods.

Contribution

It introduces new methods for sampling and generating realistic true-false datasets from tabular and QA data, challenging prior synthetic dataset-based findings.

Findings

Partial validation of previous results

Generalization to LLM-generated datasets is difficult

Provides practical guidelines for factuality evaluation

Abstract

Factual hallucinations are a major challenge for Large Language Models (LLMs). They undermine reliability and user trust by generating inaccurate or fabricated content. Recent studies suggest that when generating false statements, the internal states of LLMs encode information about truthfulness. However, these studies often rely on synthetic datasets that lack realism, which limits generalization when evaluating the factual accuracy of text generated by the model itself. In this paper, we challenge the findings of previous work by investigating truthfulness encoding capabilities, leading to the generation of a more realistic and challenging dataset. Specifically, we extend previous work by introducing: (1) a strategy for sampling plausible true-false factoid sentences from tabular data and (2) a procedure for generating realistic, LLM-dependent true-false datasets from Question Answering collections. Our analysis of two open-source LLMs reveals that while the findings from previous studies are partially validated, generalization to LLM-generated datasets remains challenging. This study lays the groundwork for future research on factuality in LLMs and offers practical guidelines for more effective evaluation.