Comparable Demonstrations are Important in In-Context Learning: A Novel Perspective on Demonstration Selection

TL;DR

This paper introduces Comparable Demonstrations (CDs) in In-Context Learning to reduce demonstration bias by emphasizing inter-demonstration relationships, leading to improved performance especially out-of-distribution.

Contribution

It proposes a novel demonstration selection method using minimally edited comparable demonstrations to mitigate bias in ICL.

Findings

Demonstration bias exists in LLMs.

CDs significantly reduce demonstration bias.

CDs improve ICL performance, especially out-of-distribution.

Abstract

In-Context Learning (ICL) is an important paradigm for adapting Large Language Models (LLMs) to downstream tasks through a few demonstrations. Despite the great success of ICL, the limitation of the demonstration number may lead to demonstration bias, i.e. the input-label mapping induced by LLMs misunderstands the task's essence. Inspired by human experience, we attempt to mitigate such bias through the perspective of the inter-demonstration relationship. Specifically, we construct Comparable Demonstrations (CDs) by minimally editing the texts to flip the corresponding labels, in order to highlight the task's essence and eliminate potential spurious correlations through the inter-demonstration comparison. Through a series of experiments on CDs, we find that (1) demonstration bias does exist in LLMs, and CDs can significantly reduce such bias; (2) CDs exhibit good performance in ICL, especially in out-of-distribution scenarios. In summary, this study explores the ICL mechanisms from a novel perspective, providing a deeper insight into the demonstration selection strategy for ICL.