UCS: Estimating Unseen Coverage for Improved In-Context Learning

TL;DR

UCS is a training-free method for selecting demonstrations in in-context learning that estimates the coverage of latent clusters to improve model performance.

Contribution

It introduces a novel coverage-based, training-free demonstration selection method using latent clusters and a Smoothed Good–Turing estimator, enhancing ICL accuracy.

Findings

UCS improves ICL accuracy by 2-6% across multiple benchmarks.

UCS provides insights into task- and model-level latent cluster distributions.

Combining UCS with existing baselines consistently enhances performance.

Abstract

In-context learning (ICL) performance depends critically on which demonstrations are placed in the prompt, yet most existing selectors prioritize heuristic notions of relevance or diversity and provide limited insight into the coverage of a demonstration set. We propose Unseen Coverage Selection (UKS), a training-free, subset-level coverage prior motivated by the principle that a good demonstration set should expose the model to latent cluster unrevealed by the currently selected subset. UCS operationalizes this idea by (1) inducing discrete latent clusters from model-consistent embeddings and (2) estimating the number of unrevealed clusters within a candidate subset via a Smoothed Good--Turing estimator from its empirical frequency spectrum. Unlike previous selection methods, UCS is coverage-based and training-free, and can be seamlessly combined with both query-dependent and query-independent selection baselines via a simple regularized objective. Experiments on multiple intent-classification and reasoning benchmarks with frontier Large Language Models show that augmenting strong baselines with UCS consistently improves ICL accuracy by up to 2-6% under the same selection budget, while also yielding insights into task- and model-level latent cluster distributions. Code is available at https://github.com/Raina-Xin/UCS.