Faster LLM Inference via Sequential Monte Carlo

TL;DR

This paper introduces SMC-SD, a novel inference method for large language models that improves speed by replacing rejection sampling with importance-weighted resampling, maintaining accuracy while significantly accelerating inference.

Contribution

The paper proposes sequential Monte Carlo speculative decoding (SMC-SD), a new approximate inference scheme that enhances inference speed without sacrificing accuracy, leveraging importance sampling and parallel computation.

Findings

SMC-SD achieves 2.36x speed-up over speculative decoding.

SMC-SD achieves 5.2x speed-up over autoregressive decoding.

SMC-SD maintains within 3% accuracy of the target model.

Abstract

Speculative decoding (SD) accelerates language model inference by drafting tokens from a cheap proposal model and verifying them against an expensive target model via rejection sampling. Because rejection truncates the draft block at the first error, throughput degrades when draft and target diverge. Rather than rejecting draft tokens outright, we propose to reweight them. To this end, we introduce sequential Monte Carlo speculative decoding (SMC-SD), which replaces token-level rejection with importance-weighted resampling over a population of draft particles. SMC-SD is a principled approximate inference scheme that trades exactness for additional speed, while preserving theoretical bounds on its per-step approximation error. Because LLM inference is memory bandwidth-bound, the arithmetic needed to draft particles and to score them in parallel comes nearly for free -- SMC-SD uses idle compute to turn verification into a vectorized, fixed-size operation with no rollback. Empirically, SMC-SD achieves 2.36x speed-up over speculative decoding and a 5.2x speed-up over autoregressive decoding, while remaining within 3% of the target model's accuracy on reasoning, instruction-following, and coding benchmarks.