Training-free Dropout Sampling for Semantic Token Acceptance in Speculative Decoding

TL;DR

DropMatch is a training-free, training-free, and calibration-free method for speculative decoding that improves token acceptance and inference speedup in large language models by matching draft tokens to the target model's predictive distribution using Monte Carlo dropout.

Contribution

It introduces DropMatch, a novel, training-free approach that enhances speculative decoding by matching draft tokens to the target model's distribution without requiring model training or calibration.

Findings

Increases acceptance length in decoding.

Achieves 1.09x to 1.33x speedup over baseline.

Operates without model training or calibration.

Abstract

Speculative decoding accelerates large language model inference by proposing tokens with a lightweight draft model and selectively accepting them using a target model. This work introduces DropMatch, a novel approach that matches draft tokens to the predictive distribution of the target model via Monte Carlo dropout applied exclusively to the LM head, enabling sampling-based acceptance decisions. By generating multiple decoding paths, our method forms an empirical token distribution against which draft tokens are evaluated for consistency. This acceptance mechanism enables the model to adaptively control the size of decoding paths under an appropriate dropout probability, preventing substantial distortion of the target model predictive distribution. The proposed method operates in a training-free, data-free, and calibration-free manner, requires no architectural modification to pretrained models, and can be orthogonally integrated with a wide range of existing speculative decoding and inference acceleration techniques. Experiments across multiple benchmarks demonstrate that our approach increases acceptance length while maintaining competitive task performance, yielding inference speedups ranging from 1.09x to 1.33x over the standard baseline, and up to an additional 1.09x speedup when applied on top of EAGLE3.