BitLM: Unlocking Multi-Token Language Generation with Bitwise Continuous Diffusion

TL;DR

BitLM introduces a binary-encoded, diffusion-based language model that generates multiple tokens simultaneously, enhancing efficiency and speed while maintaining causal language modeling principles.

Contribution

It presents a novel approach combining binary token encoding with diffusion models to enable parallel multi-token generation without losing causal structure.

Findings

Achieves faster inference compared to traditional autoregressive models.

Maintains causal language modeling while generating multiple tokens in parallel.

Demonstrates improved efficiency and potential for next-generation language models.

Abstract

Autoregressive language models generate text one token at a time, yet natural language is inherently structured in multi-token units, including phrases, n-grams, and collocations that carry meaning jointly. This one-token bottleneck limits both the expressiveness of the model during pre-training and its throughput at inference time. Existing remedies such as speculative decoding or diffusion-based language models either leave the underlying bottleneck intact or sacrifice the causal structure essential to language modeling. We propose BitLM, a language model that represents each token as a fixed-length binary code and employs a lightweight diffusion head to denoise multiple tokens in parallel within each block. Crucially, BitLM preserves left-to-right causal attention across blocks while making joint lexical decisions within each block, combining the reliability of autoregressive modeling with the parallelism of iterative refinement. By replacing the large-vocabulary softmax with bitwise denoising, BitLM reframes token generation as iterative commitment in a compact binary space, enabling more efficient pre-training and substantially faster inference without altering the causal foundation that makes language models effective. Our results demonstrate that the one-token-at-a-time paradigm is not a fundamental requirement but an interface choice, and that changing it can yield a stronger and faster language model. We hope BitLM points toward a promising direction for next-generation language model architectures.