CASCADE: Context-Aware Relaxation for Speculative Image Decoding

TL;DR

CASCADE introduces a novel context-aware relaxation method for speculative image decoding, leveraging redundancies in model representations to significantly accelerate image synthesis without quality loss.

Contribution

It formalizes properties like semantic interchangeability and convergence to enable principled acceptance relaxation in speculative decoding, improving efficiency without extra training.

Findings

Achieves up to 3.6x speedup in image decoding.

Maintains image quality and fidelity during acceleration.

Demonstrates effectiveness across multiple models and architectures.

Abstract

Autoregressive generation is a powerful approach for high-fidelity image synthesis, but it remains computationally demanding and slow even on the most advanced accelerators. While speculative decoding has been explored to mitigate this bottleneck, existing approaches fail to achieve efficiency gains comparable to those observed in text generation. A key limitation is the target model's high uncertainty during image generation, which leads to high draft token rejection rates. In this work, we identify previously overlooked patterns in the target model's behavior that emerge naturally in tree-based speculative decoding. Specifically, we formalize two properties, semantic interchangeability and convergence, arising from the redundancies in the target model's hidden state representations. By capturing these redundancies across the depth and breadth of the predicted token tree, our method identifies principled opportunities for acceptance relaxation without requiring additional training. Additionally, we enhance standalone drafter performance by injecting the redundancy signals from the target model into drafter training with minimal modification. We evaluate our approach across multiple text-to-image models and drafter architectures. Results show that CASCADE achieves state-of-the-art speedups for drafter-based speculative decoding, with up to 3.6x acceleration, while maintaining image quality and text-prompt fidelity.