Compositional Exemplars for In-context Learning

TL;DR

This paper introduces CEIL, a novel method for selecting in-context examples using DPPs and contrastive learning, significantly improving large language models' in-context learning across diverse NLP tasks.

Contribution

We propose CEIL, a new subset selection approach for in-context learning that models interactions with DPPs and optimizes via contrastive learning, achieving state-of-the-art results.

Findings

CEIL outperforms existing selection methods across 12 NLP datasets.

CEIL demonstrates strong transferability and compositionality.

The approach achieves state-of-the-art performance in in-context learning.

Abstract

Large pretrained language models (LMs) have shown impressive In-Context Learning (ICL) ability, where the model learns to do an unseen task via a prompt consisting of input-output examples as the demonstration, without any parameter updates. The performance of ICL is highly dominated by the quality of the selected in-context examples. However, previous selection methods are mostly based on simple heuristics, leading to sub-optimal performance. In this work, we formulate in-context example selection as a subset selection problem. We propose CEIL (Compositional Exemplars for In-context Learning), which is instantiated by Determinantal Point Processes (DPPs) to model the interaction between the given input and in-context examples, and optimized through a carefully-designed contrastive learning objective to obtain preference from LMs. We validate CEIL on 12 classification and generation datasets from 7 distinct NLP tasks, including sentiment analysis, paraphrase detection, natural language inference, commonsense reasoning, open-domain question answering, code generation, and semantic parsing. Extensive experiments demonstrate not only the state-of-the-art performance but also the transferability and compositionality of CEIL, shedding new light on effective and efficient in-context learning. Our code is released at https://github.com/HKUNLP/icl-ceil.