Component-Aware Self-Speculative Decoding in Hybrid Language Models

TL;DR

This paper introduces component-aware self-speculative decoding for hybrid language models, exploiting architectural heterogeneity to improve decoding efficiency and acceptance rates.

Contribution

It is the first method to leverage internal architectural heterogeneity in hybrid models for self-speculative decoding, demonstrating significant performance differences based on architecture.

Findings

Parallel hybrids achieve acceptance rate alpha=0.68 at draft length 2.

Sequential hybrids achieve acceptance rate alpha=0.038 at draft length 2.

Perplexity degradation predicts speculative viability without decoding.

Abstract

Speculative decoding accelerates autoregressive inference by drafting candidate tokens with a fast model and verifying them in parallel with the target. Self-speculative methods avoid the need for an external drafter but have been studied exclusively in homogeneous Transformer architectures. We introduce component-aware self-speculative decoding, the first method to exploit the internal architectural heterogeneity of hybrid language models, isolating the SSM/linear-attention subgraph as a zero-cost internal draft. We evaluate this on two architecturally distinct hybrid families: Falcon-H1 (parallel: Mamba-2 + attention per layer) and Qwen3.5 (sequential: interleaved linear and attention layers), with a pure Transformer control (Qwen2.5). Parallel hybrids achieve acceptance rates of alpha = 0.68 at draft length k=2 under greedy decoding, while sequential hybrids yield only alpha = 0.038 -- an 18x gap attributable to how each architecture integrates its components. The property is scale-invariant: Falcon-H1 at 3B reproduces the rates observed at 0.5B. We further show that perplexity degradation from a companion ablation study predicts speculative viability without running speculative decoding: a 3.15x ratio (Falcon) maps to alpha = 0.37 at k=4, while 81.96x (Qwen) maps to alpha = 0.019. For sequential hybrids, generic LayerSkip achieves 12x higher acceptance rates than the component-aware strategy. The composition pattern of hybrid models -- not merely the presence of alternative components -- determines whether component-level self-speculation is viable.