SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning

TL;DR

SpecVLM introduces a verifier-guided token pruning framework that significantly accelerates video LLM decoding by pruning up to 90% of tokens without losing accuracy, leveraging staged pruning and low sensitivity of speculation.

Contribution

It presents a training-free speculative decoding method with staged token pruning for Vid-LLMs, achieving lossless acceleration and robustness.

Findings

Up to 2.68× decoding speedup on benchmark models.

Prunes up to 90% of video tokens without accuracy loss.

Effective across multiple video understanding benchmarks.

Abstract

Video large language models (Vid-LLMs) have shown strong capabilities in understanding video content. However, their reliance on dense video token representations introduces substantial memory and computational overhead in both prefilling and decoding. To mitigate the information loss of recent video token reduction methods and accelerate the decoding stage of Vid-LLMs losslessly, we introduce SpecVLM, a training-free speculative decoding (SD) framework tailored for Vid-LLMs that incorporates staged video token pruning. Building on our novel finding that the draft model's speculation exhibits low sensitivity to video token pruning, SpecVLM prunes up to 90% of video tokens to enable efficient speculation without sacrificing accuracy. To achieve this, we performs a two-stage pruning process: Stage I selects highly informative tokens guided by attention signals from the verifier (target model), while Stage II prunes remaining redundant ones in a spatially uniform manner. Extensive experiments on four video understanding benchmarks demonstrate the effectiveness and robustness of SpecVLM, which achieves up to 2.68$\times$ decoding speedup for LLaVA-OneVision-72B and 2.11$\times$ speedup for Qwen2.5-VL-32B. Code is available at https://github.com/zju-jiyicheng/SpecVLM.