Interpretable Probability Estimation with LLMs via Shapley Reconstruction

TL;DR

This paper introduces PRISM, a framework that enhances the transparency and accuracy of probability estimates from Large Language Models by decomposing predictions into input factors using Shapley values, applicable across various domains.

Contribution

PRISM is a novel method that reconstructs LLM probability estimates with improved calibration and interpretability through Shapley value decomposition.

Findings

PRISM outperforms direct prompting in predictive accuracy.

PRISM provides transparent insights into factor contributions.

Applicable across finance, healthcare, and agriculture domains.

Abstract

Large Language Models (LLMs) demonstrate potential to estimate the probability of uncertain events, by leveraging their extensive knowledge and reasoning capabilities. This ability can be applied to support intelligent decision-making across diverse fields, such as financial forecasting and preventive healthcare. However, directly prompting LLMs for probability estimation faces significant challenges: their outputs are often noisy, and the underlying predicting process is opaque. In this paper, we propose PRISM: Probability Reconstruction via Shapley Measures, a framework that brings transparency and precision to LLM-based probability estimation. PRISM decomposes an LLM's prediction by quantifying the marginal contribution of each input factor using Shapley values. These factor-level contributions are then aggregated to reconstruct a calibrated final estimate. In our experiments, we demonstrate PRISM improves predictive accuracy over direct prompting and other baselines, across multiple domains including finance, healthcare, and agriculture. Beyond performance, PRISM provides a transparent prediction pipeline: our case studies visualize how individual factors shape the final estimate, helping build trust in LLM-based decision support systems.