DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understanding

TL;DR

DynTok introduces a dynamic token compression method for video understanding that significantly reduces token count while maintaining performance, enabling more efficient processing of long videos.

Contribution

It proposes a novel adaptive compression strategy that merges visual tokens based on information density, reducing computational load in video modeling.

Findings

Reduces visual tokens to 44.4% of original size

Maintains comparable performance with fewer tokens

Achieves 65.3% on Video-MME and 72.5% on MLVU

Abstract

Typical video modeling methods, such as LLava, represent videos as sequences of visual tokens, which are then processed by the LLM backbone for effective video understanding. However, this approach leads to a massive number of visual tokens, especially for long videos. A practical solution is to first extract relevant visual information from the large visual context before feeding it into the LLM backbone, thereby reducing computational overhead. In this work, we introduce DynTok, a novel \textbf{Dyn}amic video \textbf{Tok}en compression strategy. DynTok adaptively splits visual tokens into groups and merges them within each group, achieving high compression in regions with low information density while preserving essential content. Our method reduces the number of tokens to 44.4% of the original size while maintaining comparable performance. It further benefits from increasing the number of video frames and achieves 65.3% on Video-MME and 72.5% on MLVU. By applying this simple yet effective compression method, we expose the redundancy in video token representations and offer insights for designing more efficient video modeling techniques.