Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding

TL;DR

This paper introduces a vocabulary trimming method for draft models in speculative decoding, optimizing the trade-off between coverage and latency to improve inference speed for large language models.

Contribution

It formulates vocabulary selection as a constrained optimization problem and uses a Tree-structured Parzen Estimator to efficiently find the optimal balance between coverage and latency.

Findings

Achieves up to 97% vocabulary reduction with high coverage

Reduces draft latency by up to 16% and improves throughput by 20% on domain-specific tasks

Gains up to 6.7% throughput on diverse out-of-distribution tasks

Abstract

Speculative decoding accelerates inference for Large Language Models by using a lightweight draft model to propose candidate tokens that are verified in parallel by a larger target model. Prior work shows that the draft model often dominates speculative decoding latency, since it generates tokens sequentially and incurs high cost from its language modeling head as vocabulary size grows. This exposes a fundamental trade-off in draft model design: larger vocabularies improve token coverage and agreement with the target model, but incur higher draft latency, while smaller vocabularies reduce latency at the risk of missing tokens required for accurate draft generation. We address this trade-off through vocabulary trimming for draft models, motivated by the observation that domain-specific workloads use only a small fraction of the full vocabulary. We cast draft vocabulary selection as a constrained optimization problem that balances token coverage and draft latency. Coverage is computed over assistant responses in the training data, while latency is estimated using architecture-aware FLOPs that capture the cost of the language modeling head as a function of vocabulary size. We optimize a utility function with a Tree-structured Parzen Estimator to efficiently explore the coverage-latency Pareto frontier under a minimum coverage constraint. Experiments show improved speculative decoding throughput while reducing draft vocabularies by up to 97% with high coverage. On domain-specific tasks, we achieve up to 16% latency reduction and 20% throughput improvement, and up to 6.7% throughput gains on diverse out-of-distribution tasks.