Efficient Training-Free Multi-Token Prediction via Embedding-Space Probing

TL;DR

This paper introduces a training-free method for multi-token prediction in large language models that leverages embedding-space probing to enable parallel token generation, significantly improving efficiency and performance.

Contribution

The authors propose a novel, training-free multi-token prediction approach using embedding-space probing, which enhances decoding efficiency without modifying the model or requiring auxiliary models.

Findings

Outperforms existing training-free baselines on multiple benchmarks.

Increases acceptance length by approximately 12% on LLaMA3 and 8-12% on Qwen3.

Achieves throughput gains of up to 19%.

Abstract

Large language models (LLMs) exhibit latent multi-token prediction (MTP) capabilities despite being trained solely for next-token generation. We propose a simple, training-free MTP approach that probes an LLM using on-the-fly mask tokens drawn from its embedding space, enabling parallel prediction of future tokens without modifying model weights or relying on auxiliary draft models. Our method constructs a speculative token tree by sampling top-K candidates from mask-token logits and applies a lightweight pruning strategy to retain high-probability continuations. During decoding, candidate predictions are verified in parallel, resulting in lossless generation while substantially reducing the number of model calls and improving token throughput. Across benchmarks, our probing-based MTP consistently outperforms existing training-free baselines, increasing acceptance length by approximately 12\% on LLaMA3 and 8--12\% on Qwen3, and achieving throughput gains of up to 15--19\%. Finally, we provide theoretical insights and empirical evidence showing that decoder layers naturally align mask-token representations with next-token states, enabling accurate multi-step prediction without retraining or auxiliary models.