Enhancing In-Context Learning via Implicit Demonstration Augmentation

TL;DR

This paper introduces a novel demonstration augmentation method that enhances in-context learning by enriching demonstration representations and theoretically linking augmentation to logit calibration, leading to improved accuracy and stability.

Contribution

It proposes a new augmentation approach based on deep feature distribution and provides theoretical insights connecting augmentation to logit calibration, improving ICL performance.

Findings

Significant accuracy improvements across diverse PLMs and tasks.

Reduction in performance variance among demonstrations and permutations.

Effective handling of imbalanced class distributions.

Abstract

The emergence of in-context learning (ICL) enables large pre-trained language models (PLMs) to make predictions for unseen inputs without updating parameters. Despite its potential, ICL's effectiveness heavily relies on the quality, quantity, and permutation of demonstrations, commonly leading to suboptimal and unstable performance. In this paper, we tackle this challenge for the first time from the perspective of demonstration augmentation. Specifically, we start with enriching representations of demonstrations by leveraging their deep feature distribution. We then theoretically reveal that when the number of augmented copies approaches infinity, the augmentation is approximately equal to a novel logit calibration mechanism integrated with specific statistical properties. This insight results in a simple yet highly efficient method that significantly improves the average and worst-case accuracy across diverse PLMs and tasks. Moreover, our method effectively reduces performance variance among varying demonstrations, permutations, and templates, and displays the capability to address imbalanced class distributions.