The Curious Case of Factuality Finetuning: Models' Internal Beliefs Can Improve Factuality

TL;DR

This paper investigates how finetuning language models on data filtered by their own internal beliefs can more effectively reduce hallucinations than traditional methods using gold factual data.

Contribution

It reveals that finetuning on model-generated data filtered by the model's own judgments improves factuality more than using gold data, offering a new approach to mitigate hallucinations.

Findings

Finetuning on model-generated data filtered by internal judgments enhances factuality.

Gold data alone is less effective than model-filtered data for reducing hallucinations.

Model's internal beliefs serve as a valuable signal for factuality across multiple domains.

Abstract

Language models are prone to hallucination - generating text that is factually incorrect. Finetuning models on high-quality factual information can potentially reduce hallucination, but concerns remain; obtaining factual gold data can be expensive and training on correct but unfamiliar data may potentially lead to even more downstream hallucination. What data should practitioners finetune on to mitigate hallucinations in language models? In this work, we study the relationship between the factuality of finetuning data and the prevalence of hallucinations in long-form generation tasks. Counterintuitively, we find that finetuning on factual gold data is not as helpful as finetuning on model-generated data that models believe to be factual. Next, we evaluate filtering strategies applied on both factual gold data and model-generated data, and find that finetuning on model-generated data that is filtered by models' own internal judgments often leads to better overall factuality compared to other configurations: training on gold data filtered by models' judgments, training on gold data alone, or training on model-generated data that is supported by gold data. These factuality improvements transfer across three domains we study, suggesting that a models' own beliefs can provide a powerful signal for factuality.