In-Context Learning with Hypothesis-Class Guidance

TL;DR

This paper introduces ICL-HCG, a synthetic data model for in-context learning that incorporates hypothesis class descriptions, demonstrating that models like Transformers can generalize to new hypotheses and classes with improved accuracy when guided by instructions.

Contribution

The paper proposes a novel synthetic data framework for in-context learning that includes hypothesis class guidance, enabling analysis of generalization, architecture effects, and instruction roles.

Findings

Transformers can learn ICL-HCG and generalize to unseen hypotheses and classes.

ICL-HCG with instructions outperforms ICL without instructions in accuracy.

The study explores sample complexity, data imbalance, and pretraining effects on ICL.

Abstract

Recent research has investigated the underlying mechanisms of in-context learning (ICL) both theoretically and empirically, often using data generated from simple function classes. However, the existing work often focuses on the sequence consisting solely of labeled examples, while in practice, labeled examples are typically accompanied by an instruction, providing some side information about the task. In this work, we propose ICL with hypothesis-class guidance (ICL-HCG), a novel synthetic data model for ICL where the input context consists of the literal description of a (finite) hypothesis class H and $(x,y)$ pairs from a hypothesis chosen from H. Under our framework ICL-HCG, we conduct extensive experiments to explore: (i) a variety of generalization abilities to new hypothesis classes; (ii) different model architectures; (iii) sample complexity; (iv) in-context data imbalance; (v) the role of instruction; and (vi) the effect of pretraining hypothesis diversity. As a result, we show that (a) Transformers can successfully learn ICL-HCG and generalize to unseen hypotheses and unseen hypothesis classes, and (b) compared with ICL without instruction, ICL-HCG achieves significantly higher accuracy, demonstrating the role of instructions.