Differentiable Conformal Training for LLM Reasoning Factuality

TL;DR

This paper introduces Differentiable Coherent Factuality (DCF), a novel method that enhances LLM reasoning reliability by jointly validating claims with logical dependencies, significantly improving claim retention while maintaining statistical guarantees.

Contribution

The paper presents DCF, a fully differentiable relaxation of prior conformal prediction methods, enabling learned scoring functions for better factuality validation in LLM reasoning.

Findings

DCF achieves up to 141% improvement in claim retention.

Maintains reliability guarantees while improving factual claim retention.

Demonstrates effectiveness on two benchmark reasoning datasets.

Abstract

Large Language Models (LLMs) frequently hallucinate, limiting their reliability in critical applications. Conformal Prediction (CP) addresses this by calibrating error rates on held-out data to provide statistically valid confidence guarantees. Recent work extends CP to LLM factuality to filter out risky claims, ensuring that hallucination rates remain below a user-specified level (e.g., 10%). While prior methods treat claims independently, Coherent Factuality extends to multi-step reasoning by representing outputs as dependency graphs and jointly validating claims with their logical ancestors. A key limitation is that Coherent Factuality is not differentiable, requiring hand-crafted scorers that at high reliability levels remove nearly 60% of true claims. We introduce Differentiable Coherent Factuality (DCF), a fully differentiable relaxation that enables learning improved scorers while provably recovering the original algorithm's guarantees. Experiments on two benchmark reasoning datasets demonstrate DCF achieves up to 141% improvement in claim retention while maintaining reliability guarantees, representing a significant step towards reliable conformal LLM systems.