PIAST: Rapid Prompting with In-context Augmentation for Scarce Training data

TL;DR

PIAST is a rapid automatic prompt construction method that enhances few-shot learning by efficiently selecting examples, outperforming existing methods on multiple NLP tasks with limited compute budgets.

Contribution

Introduces a fast, automatic prompt augmentation algorithm using Monte Carlo Shapley estimation, improving prompt quality with minimal compute for scarce data scenarios.

Findings

Outperforms existing automatic prompting methods on text simplification and GSM8K.

Achieves second best results on classification and summarization tasks.

Sets new state of the art with increased compute budget on multiple NLP tasks.

Abstract

LLMs are highly sensitive to prompt design, but handcrafting effective prompts is difficult and often requires intricate crafting of few-shot examples. We propose a fast automatic prompt construction algorithm that augments human instructions by generating a small set of few shot examples. Our method iteratively replaces/drops/keeps few-shot examples using Monte Carlo Shapley estimation of example utility. For faster execution, we use aggressive subsampling and a replay buffer for faster evaluations. Our method can be run using different compute time budgets. On a limited budget, we outperform existing automatic prompting methods on text simplification and GSM8K and obtain second best results on classification and summarization. With an extended, but still modest compute budget we set a new state of the art among automatic prompting methods on classification, simplification and GSM8K. Our results show that carefully constructed examples, rather than exhaustive instruction search, are the dominant lever for fast and data efficient prompt engineering. Our code is available at https://github.com/Batorskq/PIAST.