Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention

TL;DR

This paper introduces a unified spatiotemporal token compression method for Video-LLMs that significantly reduces computational costs while maintaining high performance, by globally selecting and merging tokens based on attention and semantic relevance.

Contribution

It proposes a novel unified token selection and merging strategy that operates without retraining, improving efficiency in Video-LLMs at ultra-low token retention levels.

Findings

Retaining about 2% of tokens preserves over 90% of baseline performance.

Reduces FLOPs to approximately 2.6% of original.

Decreases inference latency and memory usage across various models.

Abstract

Video large language models (Video-LLMs) face high computational costs due to large volumes of visual tokens. Existing token compression methods typically adopt a two-stage spatiotemporal compression strategy, relying on stage-specific metrics and an implicit assumption of spatiotemporal separability. Under extremely low retention ratios, however, such approaches often result in unbalanced allocation and loss of visual evidence essential for question answering. We reformulate token compression as a spatiotemporal allocation task within a global token retention pool. We propose a unified selection mechanism that integrates attention weights and semantic similarity to globally select tokens with high contribution and low redundancy. Unselected tokens are merged via clustering and refilled, preserving information integrity. Inside the LLM, we further introduce text-aware merging to perform secondary compression based on query relevance. Without requiring retraining, our method serves as a plug-and-play module compatible with existing Video-LLMs. Experiments show that retaining only about 2% of visual tokens preserves 90.1% of baseline performance across multiple benchmarks, while reducing FLOPs to roughly 2.6%. These benefits generalize across diverse backbones, decreasing end-to-end inference latency and memory consumption. Our unified spatiotemporal token compression strategy establishes the state-of-the-art in video understanding under ultra-low token retention.