FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness

TL;DR

FAITH is a post-training framework that improves LLM factuality by integrating trustworthiness and honestness signals, external knowledge retrieval, and a reward-based fine-tuning process.

Contribution

It introduces a novel natural-language based approach to align LLMs' internal trust and honesty with external knowledge, enhancing factual accuracy.

Findings

FAITH improves factual accuracy on four benchmarks.

The retrieval module increases consistency between internal and external knowledge.

Reward-based fine-tuning enhances truthfulness of LLM outputs.

Abstract

Large Language Models (LLMs) can generate factually inaccurate content even if they have corresponding knowledge, which critically undermines their reliability. Existing approaches attempt to mitigate this by incorporating uncertainty in QA prompt during training, but these numerical scores lack the semantic richness for LLM to properly understand its internal states of trustworthiness and honestness, leading to insufficient factuality alignment. We introduce FAITH (Factuality Alignment through Integrating Trustworthiness and Honestness), a post-training framework for factuality alignment that integrates natural-language uncertainty signals with external knowledge. Specifically, we augment training datasets by computing confidence scores and semantic entropy from LLM outputs and mapping them into a knowledge state quadrant that describes the model's internal knowledge possession (trustworthiness) and answering behaviors (honestness) in natural language. Based on this enhanced data, we design a reward function that considers both correctness and uncertainty signals, and fine-tune the LLM using the Proximal Policy Optimization (PPO) algorithm. To further mitigate weakly grounded responses, we design a retrieval-augmented module that retrieves relevant external passages, improving the consistency between internal and external knowledge representations. Extensive experiments on four knowledge-intensive benchmarks demonstrate that FAITH enhances the factual accuracy and truthfulness of LLMs.