How are Prompts Different in Terms of Sensitivity?

TL;DR

This paper introduces a sensitivity-based analysis of prompts in in-context learning, showing sensitivity correlates with performance and proposing a sensitivity-aware decoding method to improve results especially with limited input information.

Contribution

It presents a novel sensitivity analysis framework for prompts, linking sensitivity to model accuracy, and introduces a sensitivity-aware decoding technique for enhanced performance.

Findings

Sensitivity negatively correlates with model accuracy.

Gradient-based saliency scores reveal prompt influence on token relevance.

Sensitivity-aware decoding improves performance with scarce input data.

Abstract

In-context learning (ICL) has become one of the most popular learning paradigms. While there is a growing body of literature focusing on prompt engineering, there is a lack of systematic analysis comparing the effects of prompts across different models and tasks. To address this gap, we present a comprehensive prompt analysis based on the sensitivity of a function. Our analysis reveals that sensitivity is an unsupervised proxy for model performance, as it exhibits a strong negative correlation with accuracy. We use gradient-based saliency scores to empirically demonstrate how different prompts affect the relevance of input tokens to the output, resulting in different levels of sensitivity. Furthermore, we introduce sensitivity-aware decoding which incorporates sensitivity estimation as a penalty term in the standard greedy decoding. We show that this approach is particularly helpful when information in the input is scarce. Our work provides a fresh perspective on the analysis of prompts, and contributes to a better understanding of the mechanism of ICL.