Uncertainty Quantification and Decomposition for LLM-based Recommendation

TL;DR

This paper introduces a framework for quantifying and decomposing uncertainty in LLM-based recommendations, improving reliability assessment and enabling uncertainty-aware prompting to enhance recommendation quality.

Contribution

It presents a novel method to estimate and analyze uncertainty in LLM recommendations, including decomposition into recommendation and prompt uncertainties, with practical improvements.

Findings

Predictive uncertainty correlates with recommendation reliability.

Decomposed uncertainty reveals primary sources of unreliability.

Uncertainty-aware prompting reduces predictive uncertainty and improves recommendations.

Abstract

Despite the widespread adoption of large language models (LLMs) for recommendation, we demonstrate that LLMs often exhibit uncertainty in their recommendations. To ensure the trustworthy use of LLMs in generating recommendations, we emphasize the importance of assessing the reliability of recommendations generated by LLMs. We start by introducing a novel framework for estimating the predictive uncertainty to quantitatively measure the reliability of LLM-based recommendations. We further propose to decompose the predictive uncertainty into recommendation uncertainty and prompt uncertainty, enabling in-depth analyses of the primary source of uncertainty. Through extensive experiments, we (1) demonstrate predictive uncertainty effectively indicates the reliability of LLM-based recommendations, (2) investigate the origins of uncertainty with decomposed uncertainty measures, and (3) propose uncertainty-aware prompting for a lower predictive uncertainty and enhanced recommendation. Our source code and model weights are available at https://github.com/WonbinKweon/UNC_LLM_REC_WWW2025