Rethinking Label Consistency of In-Context Learning: An Implicit Transductive Label Propagation Perspective

TL;DR

This paper reinterprets in-context learning (ICL) as a transductive label propagation process, proposing a new data synthesis method to select demonstrations with consistent labels, thereby improving performance across NLP benchmarks.

Contribution

It introduces a transductive label propagation perspective to ICL, along with a novel data synthesis method, TopK-SD, for selecting demonstrations with label consistency, enhancing ICL effectiveness.

Findings

TopK-SD outperforms traditional TopK sampling on multiple benchmarks.

The proposed framework links label consistency with propagation error bounds.

A new perspective on ICL mechanisms is established.

Abstract

Large language models (LLMs) perform in-context learning (ICL) with minimal supervised examples, which benefits various natural language processing (NLP) tasks. One of the critical research focus is the selection of prompt demonstrations. Current approaches typically employ retrieval models to select the top-K most semantically similar examples as demonstrations. However, we argue that existing methods are limited since the label consistency is not guaranteed during demonstration selection. Our cognition derives from the Bayesian view of ICL and our rethinking of ICL from the transductive label propagation perspective. We treat ICL as a transductive learning method and incorporate latent concepts from Bayesian view and deduce that similar demonstrations guide the concepts of query, with consistent labels serving as estimates. Based on this understanding, we establish a label propagation framework to link label consistency with propagation error bounds. To model label consistency, we propose a data synthesis method, leveraging both semantic and label information, and use TopK sampling with Synthetic Data (TopK-SD) to acquire demonstrations with consistent labels. TopK-SD outperforms original TopK sampling on multiple benchmarks. Our work provides a new perspective for understanding the working mechanisms within ICL.