MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE

TL;DR

This paper explores how speculative decoding can significantly accelerate sparse Mixture of Experts models in large language models, especially at medium batch sizes, providing theoretical insights and practical speedup results.

Contribution

It demonstrates that speculative decoding benefits MoE models more than dense models at medium batch sizes and introduces a new metric 'target efficiency' to better understand SD performance.

Findings

Up to 2.29x speedup on GPUs for MoE models.

Speculative decoding benefits increase as MoE models become sparser.

Theoretical analysis aligns with experimental results.

Abstract

Large Language Models (LLMs) have achieved remarkable success across many applications, with Mixture of Experts (MoE) models demonstrating great potential. Compared to traditional dense models, MoEs achieve better performance with less computation. Speculative decoding (SD) is a widely used technique to accelerate LLM inference without accuracy loss, but it has been considered efficient only for dense models. In this work, we first demonstrate that, under medium batch sizes, MoE surprisingly benefits more from SD than dense models. Furthermore, as MoE becomes sparser -- the prevailing trend in MoE designs -- the batch size range where SD acceleration is expected to be effective becomes broader. To quantitatively understand tradeoffs involved in SD, we develop a reliable modeling based on theoretical analyses. While current SD research primarily focuses on improving acceptance rates of algorithms, changes in workload and model architecture can still lead to degraded SD acceleration even with high acceptance rates. To address this limitation, we introduce a new metric 'target efficiency' that characterizes these effects, thus helping researchers identify system bottlenecks and understand SD acceleration more comprehensively. For scenarios like private serving, this work unveils a new perspective to speed up MoE inference, where existing solutions struggle. Experiments on different GPUs show up to 2.29x speedup for Qwen2-57B-A14B at medium batch sizes and validate our theoretical predictions.