PRISM: Parametrically Refactoring Inference for Speculative Sampling Draft Models

TL;DR

PRISM introduces an architectural innovation for large language models that decouples model capacity from inference cost, enabling faster decoding and better scalability without sacrificing draft quality.

Contribution

PRISM's disaggregated computation approach refactors draft models, outperforming existing architectures in speed and scalability for speculative sampling in LLMs.

Findings

PRISM achieves over 2.6x speedup in decoding throughput.

PRISM scales more effectively with larger data volumes.

PRISM maintains high draft quality with minimal latency.

Abstract

Large Language Models (LLMs), constrained by their auto-regressive nature, suffer from slow decoding. Speculative decoding methods have emerged as a promising solution to accelerate LLM decoding, attracting attention from both systems and AI research communities. Recently, the pursuit of better draft quality has driven a trend toward parametrically larger draft models, which inevitably introduces substantial computational overhead. While existing work attempts to balance the trade-off between prediction accuracy and compute latency, we address this fundamental dilemma through architectural innovation. We propose PRISM, which disaggregates the computation of each predictive step across different parameter sets, refactoring the computational pathways of draft models to successfully decouple model capacity from inference cost. Through extensive experiments, we demonstrate that PRISM outperforms all existing draft architectures, achieving exceptional acceptance lengths while maintaining minimal draft latency for superior end-to-end speedup. We also re-examine scaling laws with PRISM, revealing that PRISM scales more effectively with expanding data volumes than other draft architectures. Through rigorous and fair comparison, we show that PRISM boosts the decoding throughput of an already highly optimized inference engine by more than 2.6x.