SAGE: Accelerating Vision-Language Models via Entropy-Guided Adaptive Speculative Decoding

TL;DR

SAGE introduces a dynamic, entropy-guided speculative decoding framework that adapts the tree structure in real-time to improve inference speed in vision-language models without sacrificing output quality.

Contribution

It proposes a novel adaptive speculation tree mechanism based on entropy to optimize decoding efficiency in vision-language models.

Findings

Achieves up to 3.36x speedup on LLaVA-OneVision-72B

Achieves up to 3.18x speedup on Qwen2.5-VL-72B

Maintains output quality while accelerating inference

Abstract

Speculative decoding has emerged as a promising approach to accelerate inference in vision-language models (VLMs) by enabling parallel verification of multiple draft tokens. However, existing methods rely on static tree structures that remain fixed throughout the decoding process, failing to adapt to the varying prediction difficulty across generation steps. This leads to suboptimal acceptance lengths and limited speedup. In this paper, we propose SAGE, a novel framework that dynamically adjusts the speculation tree structure based on real-time prediction uncertainty. Our key insight is that output entropy serves as a natural confidence indicator with strong temporal correlation across decoding steps. SAGE constructs deeper-narrower trees for high-confidence predictions to maximize speculation depth, and shallower-wider trees for uncertain predictions to diversify exploration. SAGE improves acceptance lengths and achieves faster acceleration compared to static tree baselines. Experiments on multiple benchmarks demonstrate the effectiveness of SAGE: without any loss in output quality, it delivers up to $3.36\times$ decoding speedup for LLaVA-OneVision-72B and $3.18\times$ for Qwen2.5-VL-72B.