Conformal Language Model Reasoning with Coherent Factuality

TL;DR

This paper introduces a conformal prediction method to ensure the factual correctness and logical coherence of language model outputs in reasoning tasks, particularly in mathematical problem solving.

Contribution

It develops a novel approach using split conformal prediction on deducibility graphs to guarantee coherent factuality in language model reasoning outputs.

Findings

Achieves 90% factuality on strict criteria.

Retains 80% of claims while ensuring correctness.

Produces consistent, substantiated reasoning orderings.

Abstract

Language models are increasingly being used in important decision pipelines, so ensuring the correctness of their outputs is crucial. Recent work has proposed evaluating the "factuality" of claims decomposed from a language model generation and applying conformal prediction techniques to filter out those claims that are not factual. This can be effective for tasks such as information retrieval, where constituent claims may be evaluated in isolation for factuality, but is not appropriate for reasoning tasks, as steps of a logical argument can be evaluated for correctness only within the context of the claims that precede them. To capture this, we define "coherent factuality" and develop a conformal-prediction-based method to guarantee coherent factuality for language model outputs. Our approach applies split conformal prediction to subgraphs within a "deducibility" graph" that represents the steps of a reasoning problem. We evaluate our method on mathematical reasoning problems from the MATH and FELM datasets and find that our algorithm consistently produces correct and substantiated orderings of claims, achieving coherent factuality across target coverage levels. Moreover, we achieve 90% factuality on our stricter definition while retaining 80% or more of the original claims, highlighting the utility of our deducibility-graph-guided approach.