Ask, and it shall be given: On the Turing completeness of prompting

TL;DR

This paper provides a theoretical foundation showing that prompting enables a finite-size Transformer to simulate any computable function, establishing its Turing completeness and universality in the context of large language models.

Contribution

It is the first to theoretically demonstrate that prompting can make a finite Transformer Turing-complete and nearly as powerful as unbounded models.

Findings

Prompting makes a finite Transformer Turing-complete.

A single finite Transformer can simulate any computable function.

Prompting enables universal computation with bounded models.

Abstract

Since the success of GPT, large language models (LLMs) have been revolutionizing machine learning and have initiated the so-called LLM prompting paradigm. In the era of LLMs, people train a single general-purpose LLM and provide the LLM with different prompts to perform different tasks. However, such empirical success largely lacks theoretical understanding. Here, we present the first theoretical study on the LLM prompting paradigm to the best of our knowledge. In this work, we show that prompting is in fact Turing-complete: there exists a finite-size Transformer such that for any computable function, there exists a corresponding prompt following which the Transformer computes the function. Furthermore, we show that even though we use only a single finite-size Transformer, it can still achieve nearly the same complexity bounds as that of the class of all unbounded-size Transformers. Overall, our result reveals that prompting can enable a single finite-size Transformer to be efficiently universal, which establishes a theoretical underpinning for prompt engineering in practice.