Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer

TL;DR

This paper develops a theoretical framework for understanding in-context learning with transformers in nonlinear regression, constructing explicit nonlinear features via attention and providing finite-sample error bounds.

Contribution

It introduces a novel construction of transformer networks that realize nonlinear features, extending ICL theory beyond linear models.

Findings

Finite-sample generalization error bounds derived.

Constructed transformer networks realize polynomial and spline bases.

Theory validated on synthetic regression tasks.

Abstract

Pre-trained transformers are able to learn from examples provided as part of the prompt without any weight updates, a remarkable ability known as in-context learning (ICL). Despite its demonstrated efficacy across various domains, the theoretical understanding of ICL is still developing. Whereas most existing theory has focused on linear models, we study ICL in the nonlinear regression setting. Through the interaction mechanism in attention, we explicitly construct transformer networks to realize nonlinear features, such as polynomial or spline bases, which span a wide class of functions. Based on this construction, we establish a framework to analyze end-to-end in-context nonlinear regression with the constructed features. Our theory provides finite-sample generalization error bounds in terms of context length and training set size. We numerically validate the theory on synthetic regression tasks.