Large Language Models as General Pattern Machines

TL;DR

This paper demonstrates that large language models can perform general pattern completion tasks across various complex sequences and spatial patterns without additional training, suggesting their potential as versatile sequence modelers for applications like robotics.

Contribution

The study reveals that pre-trained LLMs can perform zero-shot pattern completion on diverse sequences and spatial data, extending their utility beyond language tasks to general AI and robotics applications.

Findings

LLMs can complete complex token sequences from PCFGs and spatial patterns in ARC.

Pattern completion proficiency persists even with randomly sampled tokens.

LLMs can be applied to robotics for sequence extrapolation and policy discovery.

Abstract

We observe that pre-trained large language models (LLMs) are capable of autoregressively completing complex token sequences -- from arbitrary ones procedurally generated by probabilistic context-free grammars (PCFG), to more rich spatial patterns found in the Abstraction and Reasoning Corpus (ARC), a general AI benchmark, prompted in the style of ASCII art. Surprisingly, pattern completion proficiency can be partially retained even when the sequences are expressed using tokens randomly sampled from the vocabulary. These results suggest that without any additional training, LLMs can serve as general sequence modelers, driven by in-context learning. In this work, we investigate how these zero-shot capabilities may be applied to problems in robotics -- from extrapolating sequences of numbers that represent states over time to complete simple motions, to least-to-most prompting of reward-conditioned trajectories that can discover and represent closed-loop policies (e.g., a stabilizing controller for CartPole). While difficult to deploy today for real systems due to latency, context size limitations, and compute costs, the approach of using LLMs to drive low-level control may provide an exciting glimpse into how the patterns among words could be transferred to actions.