Unlabeled Data Can Provably Enhance In-Context Learning of Transformers

TL;DR

This paper introduces a new framework that uses unlabeled data alongside labeled examples to improve in-context learning of transformers, supported by theoretical analysis and empirical results showing consistent performance gains.

Contribution

It provides the first theoretical analysis demonstrating how unlabeled data can provably enhance transformer-based in-context learning, especially in multi-class classification tasks.

Findings

Transformer effectively emulates EM algorithm with unlabeled data

Augmented ICL outperforms traditional few-shot ICL

Parameters converge linearly with teacher forcing

Abstract

Large language models (LLMs) exhibit impressive in-context learning (ICL) capabilities, yet the quality of their predictions is fundamentally limited by the few costly labeled demonstrations that can fit into a prompt. Meanwhile, there exist vast and continuously growing amounts of unlabeled data that may be closely related to the ICL task. How to utilize such unlabeled data to provably enhance the performance of ICL thus becomes an emerging fundamental question. In this work, we propose a novel augmented ICL framework, in which the prompt includes a small set of labeled examples alongside a block of unlabeled inputs. We focus on the multi-class linear classification setting and demonstrate that, with chain-of-thought (CoT) prompting, a multi-layer transformer can effectively emulate an expectation-maximization (EM) algorithm. This enables the transformer to implicitly extract useful information from both labeled and unlabeled data, leading to provable improvements in ICL accuracy. Moreover, we show that such a transformer can be trained via teacher forcing, with its parameters converging to the desired solution at a linear rate. Experiments demonstrate that the augmented ICL framework consistently outperforms conventional few-shot ICL, providing empirical support for our theoretical findings. To the best of our knowledge, this is the first theoretical study on the impact of unlabeled data on the ICL performance of transformers.