Exploring the Hidden Capacity of LLMs for One-Step Text Generation

TL;DR

This paper reveals that large language models can generate hundreds of tokens from just two learned embeddings in a single forward pass, indicating a native multi-token generation capability that could bypass traditional autoregressive decoding.

Contribution

It demonstrates that frozen LLMs can produce long text segments from minimal learned inputs, uncovering a new multi-token generation ability without retraining the models.

Findings

Frozen LLMs can generate hundreds of tokens from two learned embeddings.

Embeddings encode information in connected, local regions in space.

Multi-token generation may be achievable via learned encoders without retraining.

Abstract

A recent study showed that large language models (LLMs) can reconstruct surprisingly long texts - up to thousands of tokens - via autoregressive generation from just one trained input embedding. In this work, we explore whether autoregressive decoding is essential for such reconstruction. We show that frozen LLMs can generate hundreds of accurate tokens in just one token-parallel forward pass, when provided with only two learned embeddings. This reveals a surprising and underexplored multi-token generation capability of autoregressive LLMs. We examine these embeddings and characterize the information they encode. We also empirically show that, although these representations are not unique for a given text, they form connected and local regions in embedding space - suggesting the potential to train a practical encoder. The existence of such representations hints that multi-token generation may be natively accessible in off-the-shelf LLMs via a learned input encoder, eliminating heavy retraining and helping to overcome the fundamental bottleneck of autoregressive decoding while reusing already-trained models.