An Information-Theoretic Analysis of In-Context Learning

TL;DR

This paper introduces new information-theoretic tools to analyze in-context learning, providing a unified framework that characterizes error components and their decay with training data and sequence length.

Contribution

It presents a general decomposition of meta-learning error and applies it to derive new theoretical insights into in-context learning with transformers.

Findings

Error decays with the number of training sequences

Error decreases as sequence length increases

The analysis avoids contrived assumptions of prior work

Abstract

Previous theoretical results pertaining to meta-learning on sequences build on contrived assumptions and are somewhat convoluted. We introduce new information-theoretic tools that lead to an elegant and very general decomposition of error into three components: irreducible error, meta-learning error, and intra-task error. These tools unify analyses across many meta-learning challenges. To illustrate, we apply them to establish new results about in-context learning with transformers. Our theoretical results characterizes how error decays in both the number of training sequences and sequence lengths. Our results are very general; for example, they avoid contrived mixing time assumptions made by all prior results that establish decay of error with sequence length.