Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models

TL;DR

This paper introduces VidCom2, a plug-and-play framework that adaptively compresses video tokens based on frame uniqueness, significantly accelerating VideoLLMs while maintaining high accuracy.

Contribution

We propose a novel adaptive token compression framework for VideoLLMs that addresses information loss and implementation issues, improving efficiency without sacrificing performance.

Findings

Achieves 99.6% of original performance with only 25% tokens.

Reduces LLM generation latency by 70.8%.

Compatible with other token compression methods.

Abstract

Video large language models (VideoLLM) excel at video understanding, but face efficiency challenges due to the quadratic complexity of abundant visual tokens. Our systematic analysis of token compression methods for VideoLLMs reveals two critical issues: (i) overlooking distinctive visual signals across frames, leading to information loss; (ii) suffering from implementation constraints, causing incompatibility with modern architectures or efficient operators. To address these challenges, we distill three design principles for VideoLLM token compression and propose a plug-and-play inference acceleration framework "Video Compression Commander" (VidCom2). By quantifying each frame's uniqueness, VidCom2 adaptively adjusts compression intensity across frames, effectively preserving essential information while reducing redundancy in video sequences. Extensive experiments across various VideoLLMs and benchmarks demonstrate the superior performance and efficiency of our VidCom2. With only 25% visual tokens, VidCom2 achieves 99.6% of the original performance on LLaVA-OV while reducing 70.8% of the LLM generation latency. Notably, our Frame Compression Adjustment strategy is compatible with other token compression methods to further improve their performance. Our code is available at https://github.com/xuyang-liu16/VidCom2.