Clapper: Compact Learning and Video Representation in VLMs

TL;DR

Clapper introduces a novel slow-fast video representation method with TimePerceiver for efficient temporal-spatial encoding, enabling high-performance video understanding with significantly reduced visual tokens.

Contribution

The paper proposes Clapper, a new approach combining slow-fast strategies and TimePerceiver to improve long and short video modeling in VLMs with substantial token compression.

Findings

Achieves 13x token compression without accuracy loss

Outperforms existing models on VideoMME, MLVU, TempCompass benchmarks

Uses fewer than 6,000 visual tokens per video

Abstract

Current vision-language models (VLMs) have demonstrated remarkable capabilities across diverse video understanding applications. Designing VLMs for video inputs requires effectively modeling the temporal dimension (i.e. capturing dependencies across frames) and balancing the processing of short and long videos. Specifically, short videos demand preservation of fine-grained details, whereas long videos require strategic compression of visual information to handle extensive temporal contexts efficiently. However, our empirical analysis reveals a critical limitation: most existing VLMs suffer severe performance degradation in long video understanding tasks when compressing visual tokens below a quarter of their original visual tokens. To enable more effective modeling of both short and long video inputs, we propose Clapper, a method that utilizes a slow-fast strategy for video representation and introduces a novel module named TimePerceiver for efficient temporal-spatial encoding within existing VLM backbones. By using our method, we achieves 13x compression of visual tokens per frame (averaging 61 tokens/frame) without compromising QA accuracy. In our experiments, Clapper achieves 62.0% on VideoMME, 69.8% on MLVU, and 67.4% on TempCompass, all with fewer than 6,000 visual tokens per video. The code will be publicly available on the homepage.