Is Conformal Factuality for RAG-based LLMs Robust? Novel Metrics and Systematic Insights

TL;DR

This paper evaluates the robustness and usefulness of conformal factuality filtering in RAG-based LLMs, revealing limitations under distribution shifts and proposing more efficient verification methods with better utility.

Contribution

It introduces novel informativeness-aware metrics, systematically analyzes conformal factuality's limitations, and compares lightweight verifiers to LLM-based scorers for improved reliability and efficiency.

Findings

Conformal filtering has low usefulness at high factuality levels due to vacuous outputs.

Factuality guarantees are fragile under distribution shifts and distractors.

Lightweight entailment-based verifiers outperform LLM-based confidence scorers in efficiency.

Abstract

Large language models (LLMs) frequently hallucinate, limiting their reliability in knowledge-intensive applications. Retrieval-augmented generation (RAG) and conformal factuality have emerged as potential ways to address this limitation. While RAG aims to ground responses in retrieved evidence, it provides no statistical guarantee that the final output is correct. Conformal factuality filtering offers distribution-free statistical reliability by scoring and filtering atomic claims using a threshold calibrated on held-out data, however, the informativeness of the final output is not guaranteed. We systematically analyze the reliability and usefulness of conformal factuality for RAG-based LLMs across generation, scoring, calibration, robustness, and efficiency. We propose novel informativeness-aware metrics that better reflect task utility under conformal filtering. Across three benchmarks and multiple model families, we find that (i) conformal filtering suffers from low usefulness at high factuality levels due to vacuous outputs, (ii) conformal factuality guarantee is not robust to distribution shifts and distractors, highlighting the limitation that requires calibration data to closely match deployment conditions, and (iii) lightweight entailment-based verifiers match or outperform LLM-based model confidence scorers while requiring over $100\times$ fewer FLOPs. Overall, our results expose factuality-informativeness trade-offs and fragility of conformal filtering framework under distribution shifts and distractors, highlighting the need for new approaches for reliability with robustness and usefulness as key metrics, and provide actionable guidance for building RAG pipelines that are both reliable and computationally efficient.