GRaSp: Automatic Example Optimization for In-Context Learning in Low-Data Tasks

TL;DR

GRaSp is a three-stage framework that automatically optimizes in-context examples for large language models, significantly improving performance on low-data named entity recognition tasks.

Contribution

It introduces a novel three-stage method combining synthetic data generation, clustering, and genetic algorithms for in-context example optimization.

Findings

GRaSp improves NER performance over zero-shot and random baselines.

Synthetic candidate pools with diverse data are crucial for effective in-context learning.

The custom diversity-adaptive mutation enhances the genetic algorithm's effectiveness.

Abstract

In-context learning enables large language models to adapt to new tasks, but their performance is highly sensitive to the selected examples. Finding effective demonstrations is particularly difficult in domain-specific, low-data settings where high-quality examples are scarce. We propose GRaSp, a three-stage framework for automatic in-context example optimization. By first generating a large synthetic candidate pool, then structuring it with clustering and dimensionality reduction, and finally using genetic algorithms to find the optimal in-context examples, the framework shows consistent improvements on the NER task. We also introduce a custom diversity-adaptive mutation mechanism, allowing it to transition from the initial broad inter-cluster exploration to focused intra-cluster refinement as the population converges. We evaluate GRaSp on financial named entity recognition (FiNER-139), comparing synthetic and human-annotated candidate pools across pool sizes of 500 and 5000. With non-synthetic data, GRaSp achieves 45.84% micro-F1, consistently outperforming both zero-shot and random few-shot baselines. Synthetic data matches the random baseline but does not exceed it, suggesting that distributional variety in the candidate pool is critical for generalization.