Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Control

TL;DR

This paper presents a conformal prediction framework with p-value testing to improve the reliability and factual accuracy of large language models in multiple-choice question answering, ensuring statistically rigorous uncertainty quantification.

Contribution

It introduces a novel integration of conformal prediction and significance testing for LLMs in MCQA, providing provable risk control and improved trustworthiness.

Findings

Achieves user-specified empirical miscoverage rates.

Prediction set size decreases with higher risk levels, validating uncertainty metrics.

Demonstrates effectiveness on MMLU benchmarks with off-the-shelf LLMs.

Abstract

This study introduces a significance testing-enhanced conformal prediction (CP) framework to improve trustworthiness of large language models (LLMs) in multiple-choice question answering (MCQA). While LLMs have been increasingly deployed in disciplinary QA scenarios, hallucination and nonfactual generation substantially compromise response reliability. Although CP provides statistically rigorous marginal coverage guarantees for prediction sets, and significance testing offers established statistical rigor, their synergistic integration remains unexplored. To mitigate hallucination and factual inaccuracies, our framework integrates $p$-value computation with conformity scoring through self-consistency resampling of MCQA responses. This approach calculates option frequencies to address LLMs' black-box nature, subsequently constructing prediction sets via null hypothesis testing ($\mathcal{H}_0$) with empirically derived $p$-values. Evaluations on MMLU and MMLU-Pro benchmarks using off-the-shelf LLMs demonstrate: (1) The enhanced CP achieves user-specified empirical miscoverage rates; (2) Test-set average prediction set size (APSS) decreases monotonically with increasing risk levels ($\alpha$), validating APSS as an effective uncertainty metric. This work establishes a principled statistical framework for trustworthy LLM deployment in high-stakes QA applications.