MLPs Learn In-Context on Regression and Classification Tasks

TL;DR

This paper demonstrates that multi-layer perceptrons (MLPs) can learn in-context tasks, perform comparably to Transformers, and even outperform them on certain relational reasoning tasks, challenging the notion that in-context learning is unique to attention-based models.

Contribution

It reveals that MLPs can learn in-context and perform relational reasoning, expanding the understanding of architectures capable of in-context learning beyond Transformers.

Findings

MLPs can learn in-context on synthetic tasks.

MLPs and MLP-Mixer models perform comparably to Transformers under same compute.

MLPs outperform Transformers on classical relational reasoning tasks.

Abstract

In-context learning (ICL), the remarkable ability to solve a task from only input exemplars, is often assumed to be a unique hallmark of Transformer models. By examining commonly employed synthetic ICL tasks, we demonstrate that multi-layer perceptrons (MLPs) can also learn in-context. Moreover, MLPs, and the closely related MLP-Mixer models, learn in-context comparably with Transformers under the same compute budget in this setting. We further show that MLPs outperform Transformers on a series of classical tasks from psychology designed to test relational reasoning, which are closely related to in-context classification. These results underscore a need for studying in-context learning beyond attention-based architectures, while also challenging prior arguments against MLPs' ability to solve relational tasks. Altogether, our results highlight the unexpected competence of MLPs in a synthetic setting, and support the growing interest in all-MLP alternatives to Transformer architectures. It remains unclear how MLPs perform against Transformers at scale on real-world tasks, and where a performance gap may originate. We encourage further exploration of these architectures in more complex settings to better understand the potential comparative advantage of attention-based schemes.