Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior

TL;DR

DyToK is a training-free, LLM-guided dynamic token compression method that improves the efficiency of video understanding models by selectively retaining semantically rich frames, achieving significant speedups without accuracy loss.

Contribution

This work introduces DyToK, a novel attention-based, training-free token compression approach that dynamically adjusts token retention based on LLM attention, outperforming existing methods in efficiency-accuracy tradeoffs.

Findings

Achieves 4.3x faster inference speed.

Maintains accuracy across multiple VLLMs.

Compatible with existing compression methods.

Abstract

Recent advances in Video Large Language Models (VLLMs) have achieved remarkable video understanding capabilities, yet face critical efficiency bottlenecks due to quadratic computational growth with lengthy visual token sequences of long videos. While existing keyframe sampling methods can improve temporal modeling efficiency, additional computational cost is introduced before feature encoding, and the binary frame selection paradigm is found suboptimal. Therefore, in this work, we propose Dynamic Token compression via LLM-guided Keyframe prior (DyToK), a training-free paradigm that enables dynamic token compression by harnessing VLLMs' inherent attention mechanisms. Our analysis reveals that VLLM attention layers naturally encoding query-conditioned keyframe priors, by which DyToK dynamically adjusts per-frame token retention ratios, prioritizing semantically rich frames while suppressing redundancies. Extensive experiments demonstrate that DyToK achieves state-of-the-art efficiency-accuracy tradeoffs. DyToK shows plug-and-play compatibility with existing compression methods, such as VisionZip and FastV, attaining 4.3x faster inference while preserving accuracy across multiple VLLMs, such as LLaVA-OneVision and Qwen2.5-VL. Code is available at https://github.com/yu-lin-li/DyToK .