VVS: Accelerating Speculative Decoding for Visual Autoregressive Generation via Partial Verification Skipping

TL;DR

VVS introduces a novel speculative decoding framework that significantly reduces inference latency in visual autoregressive models by skipping verification steps through partial verification skipping.

Contribution

It proposes a new SD framework with verification skipping, token selection, feature caching, and step scheduling to accelerate visual AR models without quality loss.

Findings

Reduces target model forward passes by 2.8×

Maintains competitive generation quality

Outperforms conventional SD frameworks in speed-quality trade-off

Abstract

Visual autoregressive (AR) generation models have demonstrated strong potential for image generation, yet their next-token-prediction paradigm introduces considerable inference latency. Although speculative decoding (SD) has been proven effective for accelerating visual AR models, its "draft one step, then verify one step" paradigm prevents a direct reduction in the number of forward passes, limiting its acceleration potential. Motivated by the interchangeability of visual tokens, we explore verification skipping in the SD process for the first time to explicitly cut the number of target model forward passes, thereby reducing inference latency. By analyzing the characteristics of the drafting stage, we observe that verification redundancy and stale feature reusability are key factors to maintain generation quality while improving speed for verification-free steps. Inspired by these two observations, we propose a novel SD framework VVS to accelerate visual AR model via partial verification skipping, which integrates three complementary modules: (1) a verification-free token selector with dynamic truncation, (2) token-level feature caching and reuse, and (3) fine-grained skipped step scheduling. Consequently, VVS reduces the number of target model forward passes by $2.8\times$ relative to vanilla AR decoding while maintaining competitive generation quality, offering a superior speed-quality trade-off over conventional SD frameworks and revealing strong potential to reshape the SD paradigm. Our code is available at https://github.com/HyattDD/VVS.