Analyzing limits for in-context learning

TL;DR

This paper critically examines the capabilities of transformer models in in-context learning, providing empirical evidence and mathematical analysis that highlight their limitations in achieving general predictive accuracy.

Contribution

It challenges prior claims by demonstrating that transformers cannot fully implement standard learning algorithms due to architectural constraints.

Findings

Empirical evidence contradicts the idea that transformers learn standard algorithms.

Mathematical analysis shows inherent architectural limitations.

Transformers cannot attain universal predictive accuracy.

Abstract

Our paper challenges claims from prior research that transformer-based models, when learning in context, implicitly implement standard learning algorithms. We present empirical evidence inconsistent with this view and provide a mathematical analysis demonstrating that transformers cannot achieve general predictive accuracy due to inherent architectural limitations.