Meta-Sel: Efficient Demonstration Selection for In-Context Learning via Supervised Meta-Learning

TL;DR

Meta-Sel is a lightweight, supervised meta-learning method for efficient demonstration selection in in-context learning, achieving high performance without model fine-tuning or online exploration.

Contribution

It introduces Meta-Sel, a simple, interpretable scoring function trained on labeled data, and provides a comprehensive benchmark of demonstration selection methods.

Findings

Meta-Sel consistently ranks among top methods across datasets and models.

It is especially effective for smaller models, improving accuracy.

Meta-Sel requires minimal computational overhead during inference.

Abstract

Demonstration selection is a practical bottleneck in in-context learning (ICL): under a tight prompt budget, accuracy can change substantially depending on which few-shot examples are included, yet selection must remain cheap enough to run per query over large candidate pools. We propose Meta-Sel, a lightweight supervised meta-learning approach for intent classification that learns a fast, interpretable scoring function for (candidate, query) pairs from labeled training data. Meta-Sel constructs a meta-dataset by sampling pairs from the training split and using class agreement as supervision, then trains a calibrated logistic regressor on two inexpensive meta-features: TF--IDF cosine similarity and a length-compatibility ratio. At inference time, the selector performs a single vectorized scoring pass over the full candidate pool and returns the top-k demonstrations, requiring no model fine-tuning, no online exploration, and no additional LLM calls. This yields deterministic rankings and makes the selection mechanism straightforward to audit via interpretable feature weights. Beyond proposing Meta-Sel, we provide a broad empirical study of demonstration selection, benchmarking 12 methods -- spanning prompt engineering baselines, heuristic selection, reinforcement learning, and influence-based approaches -- across four intent datasets and five open-source LLMs. Across this benchmark, Meta-Sel consistently ranks among the top-performing methods, is particularly effective for smaller models where selection quality can partially compensate for limited model capacity, and maintains competitive selection-time overhead.