One size doesn't fit all: Predicting the Number of Examples for In-Context Learning

TL;DR

This paper introduces a dynamic method to predict the optimal number of examples for each data instance in in-context learning, significantly improving performance over fixed-size approaches.

Contribution

It proposes a multi-label classifier that predicts the number of examples per instance, addressing the limitations of the one-size-fits-all approach in ICL.

Findings

Outperforms standard ICL by up to 17% on text classification benchmarks.

Effectively predicts the optimal number of examples for each data instance.

Demonstrates the benefit of dynamic example selection in few-shot learning.

Abstract

In-context learning (ICL) refers to the process of adding a small number of localized examples from a training set of labelled data to an LLM's prompt with an objective to effectively control the generative process seeking to improve the downstream task performance. Existing ICL approaches use an identical number of examples (a pre-configured hyper-parameter) for each data instance. Our work alleviates the limitations of this 'one fits all' approach by dynamically predicting the number of examples for each data instance to be used in few-shot inference with LLMs. In particular, we employ a multi-label classifier, the parameters of which are fitted using a training set, where the label for each instance in this training set indicates if using a specific value of k (number of most similar examples from 0 up to a maximum value) leads to correct k-shot downstream predictions. Our experiments on a number of text classification benchmarks show that AICL substantially outperforms standard ICL by up to 17%.