Survival of the Most Influential Prompts: Efficient Black-Box Prompt Search via Clustering and Pruning
Han Zhou, Xingchen Wan, Ivan Vuli\'c, Anna Korhonen

TL;DR
This paper introduces ClaPS, a black-box prompt search method that uses clustering and pruning to focus on influential tokens, significantly improving efficiency and performance in prompt-based learning with large language models.
Contribution
The paper reveals the disproportionate influence of certain prompt tokens and proposes a novel search method that enhances efficiency by focusing on these tokens, outperforming existing approaches.
Findings
ClaPS achieves state-of-the-art results across various tasks.
The method reduces search costs significantly.
Focusing on influential tokens improves prompt search efficiency.
Abstract
Prompt-based learning has been an effective paradigm for large pretrained language models (LLM), enabling few-shot or even zero-shot learning. Black-box prompt search has received growing interest recently for its distinctive properties of gradient-free optimization, proven particularly useful and powerful for model-as-a-service usage. However, the discrete nature and the complexity of combinatorial optimization hinder the efficiency of modern black-box approaches. Despite extensive research on search algorithms, the crucial aspect of search space design and optimization has been largely overlooked. In this paper, we first conduct a sensitivity analysis by prompting LLM, revealing that only a small number of tokens exert a disproportionate amount of influence on LLM predictions. Leveraging this insight, we propose the Clustering and Pruning for Efficient Black-box Prompt Search (ClaPS),…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Speech and dialogue systems
MethodsFocus · Pruning
