Sample Efficient Demonstration Selection for In-Context Learning

TL;DR

This paper introduces CASE, a sample-efficient method for selecting exemplars in in-context learning with large language models, significantly reducing LLM evaluations while maintaining performance.

Contribution

We formulate exemplar selection as a top-m arms identification problem and propose CASE, a novel exploration strategy that reduces sample complexity and improves efficiency.

Findings

Achieves up to 7x speedup in runtime

Requires 87% fewer LLM calls

Maintains competitive performance

Abstract

The in-context learning paradigm with LLMs has been instrumental in advancing a wide range of natural language processing tasks. The selection of few-shot examples (exemplars / demonstration samples) is essential for constructing effective prompts under context-length budget constraints. In this paper, we formulate the exemplar selection task as a top-m best arms identification problem. A key challenge in this setup is the exponentially large number of arms that need to be evaluated to identify the m-best arms. We propose CASE (Challenger Arm Sampling for Exemplar selection), a novel sample-efficient selective exploration strategy that maintains a shortlist of "challenger" arms, which are current candidates for the top-m arms. In each iteration, only one of the arms from this shortlist or the current topm set is pulled, thereby reducing sample complexity and, consequently, the number of LLM evaluations. Furthermore, we model the scores of exemplar subsets (arms) using a parameterized linear scoring function, leading to stochastic linear bandits setting. CASE achieves remarkable efficiency gains of up to 7x speedup in runtime while requiring 7x fewer LLM calls (87% reduction) without sacrificing performance compared to state-of-the-art exemplar selection methods. We release our code and data at https://github.com/kiranpurohit/CASE