Active Example Selection for In-Context Learning

TL;DR

This paper introduces a reinforcement learning approach to select demonstration examples for in-context learning, improving performance on smaller models and revealing limitations on larger models.

Contribution

It formulates example selection as a sequential decision problem and develops a RL-based method to identify effective demonstration examples for in-context learning.

Findings

RL-based policies improve GPT-2 performance by 5.8% on average

Selected examples enhance GPT-3 Ada performance slightly

Larger GPT-3 models show diminishing benefits from example selection

Abstract

With a handful of demonstration examples, large-scale language models show strong capability to perform various tasks by in-context learning from these examples, without any fine-tuning. We demonstrate that in-context learning performance can be highly unstable across samples of examples, indicating the idiosyncrasies of how language models acquire information. We formulate example selection for in-context learning as a sequential decision problem, and propose a reinforcement learning algorithm for identifying generalizable policies to select demonstration examples. For GPT-2, our learned policies demonstrate strong abilities of generalizing to unseen tasks in training, with a $5.8\%$ improvement on average. Examples selected from our learned policies can even achieve a small improvement on GPT-3 Ada. However, the improvement diminishes on larger GPT-3 models, suggesting emerging capabilities of large language models.