Which Words Matter Most in Zero-Shot Prompts?
Nikta Gohari Sadr, Sangmitra Madhusudan, Hassan Sajjad, Ali Emami

TL;DR
This paper introduces the ZIP score, a systematic method to identify which words in zero-shot prompts are most influential, revealing patterns across models and tasks, and establishing a benchmark for interpretability.
Contribution
The paper presents the ZIP score, the first method to quantify individual word importance in prompts, and provides insights into word significance patterns and a benchmark for interpretability.
Findings
Nouns are most often important words in prompts.
Proprietary models align better with human intuition than open-source models.
Word importance is higher when models perform poorly.
Abstract
While zero-shot instructional prompts like "Let's think step-by-step" have revolutionized Large Language Model performance, a fundamental question remains unanswered: which specific words drive their remarkable effectiveness? We introduce the ZIP score (Zero-shot Importance of Perturbation), the first systematic method to quantify individual word importance in instructional prompts through controlled perturbations including synonym replacement, co-hyponym substitution, and strategic removal. Our analysis across four flagship models, seven widely-adopted prompts, and multiple task domains reveals four key findings: (1) Task-specific word hierarchies exist where mathematical problems prioritize "step-by-step" while reasoning tasks favor "think"; (2) Proprietary models show superior alignment with human intuitions compared to open-source alternatives; (3) Nouns dominate importance…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Text Readability and Simplification · Intelligent Tutoring Systems and Adaptive Learning
MethodsALIGN
