PrefixNLI: Detecting Factual Inconsistencies as Soon as They Arise

TL;DR

This paper introduces PrefixNLI, a new approach that detects factual inconsistencies in text prefixes during generation, improving the faithfulness of LLM outputs with a specialized model and controlled decoding.

Contribution

It generalizes entailment detection to text prefixes, develops MiniTruePrefixes for better factual inconsistency detection, and demonstrates improved summarization faithfulness.

Findings

MiniTruePrefixes outperforms baseline NLI models by 5-14 F1 points.

Integrating MiniTruePrefixes improves factual consistency in summarization.

Guided decoding with MiniTruePrefixes enables smaller models to match larger ones' faithfulness.

Abstract

Natural Language Inference (NLI) models have been used in various ways to improve the factuality of LLM outputs. This is typically done by applying an NLI model to judge whether the model output is entailed from the supposed evidence, triggering some corrective actions, such as beam reranking at inference time or RL rewards during training. While NLI models are trained to detect factual inconsistencies over complete sentences, decisions in the common autoregressive generation architecture are made for each evolving text prefix, during decoding. Addressing this setting, we generalize the entailment detection task to apply over arbitrary text prefixes, and suggest its utility for improving generation faithfulness. Providing suitable evaluation and training datasets for this task, we train MiniTruePrefixes, a novel specialized model that better detects factual inconsistencies over text prefixes, outperforming comparable baseline NLI models by 5-14 F1 points in prefix-level entailment. We further demonstrate that integrating MiniTruePrefixes into a controlled decoding framework substantially improves factual consistency in abstractive summarization. When guided by MiniTruePrefixes, LLaMA-3.2-3B-Instruct matches the faithfulness and runtime of the 8B model from the same model family, while using only half the memory.