Stateful Token Reduction for Long-Video Hybrid VLMs

TL;DR

This paper introduces a progressive token reduction method for hybrid long-video vision-language models, achieving significant speedups with minimal accuracy loss by unifying importance scoring across attention and Mamba blocks.

Contribution

It proposes a novel low-to-high progressive reduction schedule and a unified language-aware scoring mechanism for hybrid VLMs, enabling effective all-layer token reduction.

Findings

Achieves 3.8--4.2x speedup with near-baseline accuracy

Effective token importance estimation across layers

Light finetuning improves long-video benchmark performance

Abstract

Token reduction is an effective way to accelerate long-video vision-language models (VLMs), but most existing methods are designed for dense Transformers and do not directly account for hybrid architectures that interleave attention with linear-time state-space blocks (e.g., Mamba). We study query-conditioned token reduction for hybrid video VLMs and analyze reduction behavior through two properties: layerwise sparsity (how many tokens capture query-relevant information) and importance stability (whether token-importance rankings persist across depth). Although token importance is sparse within each layer, the set of important tokens changes across layers, so aggressive early pruning is unreliable. Motivated by this, we propose a low-to-high progressive reduction schedule and a unified language-aware scoring mechanism for both attention and Mamba blocks (using an implicit-attention proxy for Mamba), enabling all-layer token reduction in hybrids. Under an aggressive compression setting (retaining 25% of visual tokens), our approach delivers substantial prefilling speedups (3.8--4.2x) with near-baseline accuracy at test time, and light finetuning under reduction further improves performance on long-context video benchmarks.