MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning

TL;DR

MAPLE introduces an influence-based pseudo-labeling framework that enhances large language models' in-context learning by adaptively selecting and pseudo-labeling unlabeled data, reducing labeling costs and improving performance.

Contribution

The paper proposes MAPLE, a novel influence-based pseudo-labeling method that adaptively selects impactful unlabeled samples for in-context learning, addressing data scarcity issues.

Findings

MAPLE improves ICL performance with limited labeled data.

Pseudo-labeling impactful unlabeled samples enhances model adaptability.

Extensive experiments validate MAPLE's effectiveness across datasets.

Abstract

In-Context Learning (ICL) empowers Large Language Models (LLMs) to tackle diverse tasks by incorporating multiple input-output examples, known as demonstrations, into the input of LLMs. More recently, advancements in the expanded context windows of LLMs have led to many-shot ICL, which uses hundreds of demonstrations and outperforms few-shot ICL, which relies on fewer examples. However, this approach is often hindered by the high cost of obtaining large amounts of labeled data. To address this challenge, we propose Many-Shot Adaptive Pseudo-LabEling, namely MAPLE, a novel influence-based many-shot ICL framework that utilizes pseudo-labeled samples to compensate for the lack of label information. We first identify a subset of impactful unlabeled samples and perform pseudo-labeling on them by querying LLMs. These pseudo-labeled samples are then adaptively selected and tailored to each test query as input to improve the performance of many-shot ICL, without significant labeling costs. Extensive experiments on real-world datasets demonstrate the effectiveness of our framework, showcasing its ability to enhance LLM adaptability and performance with limited labeled data.