Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Process

TL;DR

This paper introduces LM-DPP, a language model-based determinantal point process, to select diverse and uncertain examples for in-context learning, improving few-shot learning efficiency without large labeled support sets.

Contribution

The paper proposes LM-DPP, a novel selection mechanism that considers uncertainty and diversity for optimal demonstration annotation in in-context learning.

Findings

LM-DPP effectively selects canonical examples across multiple models.

Selected subsets with low uncertainty and high diversity benefit LLM performance.

Experimental results on diverse datasets validate the approach's effectiveness.

Abstract

In-context learning (ICL) is a few-shot learning paradigm that involves learning mappings through input-output pairs and appropriately applying them to new instances. Despite the remarkable ICL capabilities demonstrated by Large Language Models (LLMs), existing works are highly dependent on large-scale labeled support sets, not always feasible in practical scenarios. To refine this approach, we focus primarily on an innovative selective annotation mechanism, which precedes the standard demonstration retrieval. We introduce the Language Model-based Determinant Point Process (LM-DPP) that simultaneously considers the uncertainty and diversity of unlabeled instances for optimal selection. Consequently, this yields a subset for annotation that strikes a trade-off between the two factors. We apply LM-DPP to various language models, including GPT-J, LlaMA, and GPT-3. Experimental results on 9 NLU and 2 Generation datasets demonstrate that LM-DPP can effectively select canonical examples. Further analysis reveals that LLMs benefit most significantly from subsets that are both low uncertainty and high diversity.