Efficient and Minimax Optimal In-context Nonparametric Regression with Transformers

TL;DR

This paper demonstrates that pretrained transformers can efficiently perform minimax optimal nonparametric regression in-context, requiring fewer parameters and pretraining sequences than previous methods.

Contribution

It introduces a transformer-based approach that approximates local polynomial estimators, achieving optimal convergence rates with significantly fewer resources.

Findings

Transformers achieve minimax optimal convergence rates in nonparametric regression.

The method requires fewer parameters and pretraining sequences than prior approaches.

Transformers implement kernel-weighted polynomial basis via gradient descent.

Abstract

We study in-context learning for nonparametric regression with $\alpha$-H\"older smooth regression functions, for some $\alpha>0$. We prove that, with $n$ in-context examples and $d$-dimensional regression covariates, a pretrained transformer with $\Theta(\log n)$ parameters and $\Omega\bigl(n^{2\alpha/(2\alpha+d)}\log^3 n\bigr)$ pretraining sequences can achieve the minimax optimal rate of convergence $O\bigl(n^{-2\alpha/(2\alpha+d)}\bigr)$ in mean squared error. Our result requires substantially fewer transformer parameters and pretraining sequences than previous results in the literature. This is achieved by showing that transformers are able to approximate local polynomial estimators efficiently by implementing a kernel-weighted polynomial basis and then running gradient descent.