Understanding the Prompt Sensitivity

TL;DR

This paper analyzes why large language models are highly sensitive to prompt wording by examining their internal behavior and deriving bounds on output differences, linking these to prompt sensitivity metrics.

Contribution

It introduces a theoretical framework modeling LLMs as multivariate functions, deriving bounds on prompt sensitivity, and identifying factors influencing this sensitivity, supported by empirical code.

Findings

LLMs disperse similar inputs rather than cluster them, increasing prompt sensitivity.

The derived upper bound correlates strongly with the PromptSensiScore metric.

Prompt templates tend to influence logits more than the questions themselves.

Abstract

Prompt sensitivity, which refers to how strongly the output of a large language model (LLM) depends on the exact wording of its input prompt, raises concerns among users about the LLM's stability and reliability. In this work, we consider LLMs as multivariate functions and perform a first-order Taylor expansion, thereby analyzing the relationship between meaning-preserving prompts, their gradients, and the log probabilities of the model's next token. We derive an upper bound on the difference between log probabilities using the Cauchy-Schwarz inequality. We show that LLMs do not internally cluster similar inputs like smaller neural networks do, but instead disperse them. This dispersing behavior leads to an excessively high upper bound on the difference of log probabilities between two meaning-preserving prompts, making it difficult to effectively reduce to 0. In our analysis, we also show which types of meaning-preserving prompt variants are more likely to introduce prompt sensitivity risks in LLMs. In addition, we demonstrate that the upper bound is strongly correlated with an existing prompt sensitivity metric, PromptSensiScore. Moreover, by analyzing the logit variance, we find that prompt templates typically exert a greater influence on logits than the questions themselves. Overall, our results provide a general interpretation for why current LLMs can be highly sensitive to prompts with the same meaning, offering crucial evidence for understanding the prompt sensitivity of LLMs. Code for experiments is available at https://github.com/ku-nlp/Understanding_the_Prompt_Sensitivity.