Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning

TL;DR

This paper investigates how learning-based demonstration selection methods work in in-context learning with large language models, revealing key factors that influence their effectiveness and proposing simplified, cost-effective selection techniques.

Contribution

It provides an empirical analysis of the mechanisms behind demonstration selection, identifying crucial similarity factors, and introduces simplified methods that improve performance without high inference costs.

Findings

Integrating multi-level text similarities improves generalization.

Using task-specific labels enhances task performance.

Simplified selection methods match complex models' effectiveness.

Abstract

Large Language Models (LLMs) have demonstrated impressive in-context learning (ICL) capabilities from few-shot demonstration exemplars. While recent learning-based demonstration selection methods have proven beneficial to ICL by choosing more useful exemplars, their underlying mechanisms are opaque, hindering efforts to address limitations such as high training costs and poor generalization across tasks. These methods generally assume the selection process captures similarities between the exemplar and the target instance, however, it remains unknown what kinds of similarities are captured and vital to performing ICL. To dive into this question, we analyze the working mechanisms of the learning-based demonstration selection methods and empirically identify two important factors related to similarity measurement: 1) The ability to integrate different levels of task-agnostic text similarities between the input of exemplars and test cases enhances generalization power across different tasks. 2) Incorporating task-specific labels when measuring the similarities significantly improves the performance on each specific task. We validate these two findings through extensive quantitative and qualitative analyses across ten datasets and various LLMs. Based on our findings, we introduce two effective yet simplified exemplar selection methods catering to task-agnostic and task-specific demands, eliminating the costly LLM inference overhead.