FactCorrector: A Graph-Inspired Approach to Long-Form Factuality Correction of Large Language Models

TL;DR

FactCorrector is a post-hoc correction method for large language models that uses structured feedback to improve factual accuracy without retraining, supported by a new benchmark dataset for evaluation.

Contribution

The paper introduces FactCorrector, a novel domain-adaptive factuality correction method leveraging structured feedback, and presents the VELI5 benchmark for rigorous evaluation.

Findings

Significantly improves factual precision of LLM outputs

Outperforms strong baseline methods

Maintains relevance while correcting factual errors

Abstract

Large language models (LLMs) are widely used in knowledge-intensive applications but often generate factually incorrect responses. A promising approach to rectify these flaws is correcting LLMs using feedback. Therefore, in this paper, we introduce FactCorrector, a new post-hoc correction method that adapts across domains without retraining and leverages structured feedback about the factuality of the original response to generate a correction. To support rigorous evaluations of factuality correction methods, we also develop the VELI5 benchmark, a novel dataset containing systematically injected factual errors and ground-truth corrections. Experiments on VELI5 and several popular long-form factuality datasets show that the FactCorrector approach significantly improves factual precision while preserving relevance, outperforming strong baselines. We release our code at https://ibm.biz/factcorrector.