Understanding the Relationship between Prompts and Response Uncertainty in Large Language Models

TL;DR

This paper explores how the informativeness of prompts influences the uncertainty of responses generated by large language models, providing insights into their reasoning process and reliability.

Contribution

It introduces a prompt-response concept model that explains the relationship between prompt informativeness and response uncertainty in LLMs.

Findings

Uncertainty decreases with increased prompt informativeness

The model explains how LLMs infer latent concepts during response generation

Experimental validation on real-world datasets supports the proposed model

Abstract

Large language models (LLMs) are widely used in decision-making, but their reliability, especially in critical tasks like healthcare, is not well-established. Therefore, understanding how LLMs reason and make decisions is crucial for their safe deployment. This paper investigates how the uncertainty of responses generated by LLMs relates to the information provided in the input prompt. Leveraging the insight that LLMs learn to infer latent concepts during pretraining, we propose a prompt-response concept model that explains how LLMs generate responses and helps understand the relationship between prompts and response uncertainty. We show that the uncertainty decreases as the prompt's informativeness increases, similar to epistemic uncertainty. Our detailed experimental results on real-world datasets validate our proposed model.