HIPPO: Accelerating Video Large Language Models Inference via Holistic-aware Parallel Speculative Decoding

TL;DR

HIPPO introduces a holistic-aware parallel speculative decoding framework that significantly accelerates video large language model inference by preserving semantic tokens and overlapping decoding phases, achieving up to 3.51x speedup.

Contribution

HIPPO's novel semantic-aware token preservation and parallel decoding strategies address previous limitations, enabling faster video-LLM inference without quality loss.

Findings

Up to 3.51x speedup over standard decoding.

Effective preservation of semantic tokens at high pruning ratios.

Validated on four video-LLMs across six benchmarks.

Abstract

Speculative decoding (SD) has emerged as a promising approach to accelerate LLM inference without sacrificing output quality. Existing SD methods tailored for video-LLMs primarily focus on pruning redundant visual tokens to mitigate the computational burden of massive visual inputs. However, existing methods do not achieve inference acceleration comparable to text-only LLMs. We observe from extensive experiments that this phenomenon mainly stems from two limitations: (i) their pruning strategies inadequately preserve visual semantic tokens, degrading draft quality and acceptance rates; (ii) even with aggressive pruning (e.g., 90% visual tokens removed), the draft model's remaining inference cost limits overall speedup. To address these limitations, we propose HIPPO, a general holistic-aware parallel speculative decoding framework. Specifically, HIPPO proposes (i) a semantic-aware token preservation method, which fuses global attention scores with local visual semantics to retain semantic information at high pruning ratios; (ii) a video parallel SD algorithm that decouples and overlaps draft generation and target verification phases. Experiments on four video-LLMs across six benchmarks demonstrate HIPPO's effectiveness, yielding up to 3.51x speedup compared to vanilla auto-regressive decoding.