SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

TL;DR

SpecForge is an open-source framework that significantly accelerates training of speculative decoding models for large language models, enabling faster inference and broader adoption in real-world applications.

Contribution

It introduces a scalable training infrastructure and high-quality draft models for speculative decoding, addressing previous limitations in model quality and training scalability.

Findings

Up to 9.9x faster training of EAGLE-3 models.

Achieved up to 4.48x inference speedup on SGLang.

Released SpecBundle, a suite of high-quality draft models.

Abstract

Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment.