TTF: Temporal Token Fusion for Efficient Video-Language Model

TL;DR

TTF is a training-free, plug-and-play method that reduces visual tokens in video-language models by exploiting temporal redundancy, significantly improving efficiency with minimal accuracy loss.

Contribution

We introduce TTF, a novel token compression framework that automatically fuses tokens based on local similarity, enabling efficient video understanding without retraining.

Findings

TTF removes about 67% of visual tokens while retaining 99.5% of accuracy.

TTF introduces only approximately 0.16 GFLOPs of overhead.

TTF is effective on Qwen3-VL-8B with a threshold of 0.70.

Abstract

Video-language models (VLMs) face rapid inference costs as visual token counts scale with video length. For example, 32 frames at $448{\times}448$ resolution already yield >8,000 visual tokens in Qwen3-VL, making LLM prefill the dominant throughput bottleneck. Existing methods often rely on global similarity or attention-guided compression, incurring offsets to their gains. We propose \textbf{Temporal Token Fusion (TTF)}, a training-free, plug-and-play pre-LLM token compression framework that exploits structured temporal redundancy in video. TTF automatically selects an anchor frame, then for each subsequent frame, performs a local window similarity search (e.g.,$3\times 3$), fusing tokens that exceed a threshold. The compressed sequence maintains positional consistency across both prefill and decoding through coordinate realignment, enabling seamless integration with existing VLM pipelines. On Qwen3-VL-8B with threshold t=0.70, TTF removes about 67\% of visual tokens while retaining 99.5\% of the baseline accuracy and introducing only ${\approx}0.16$\,GFLOPs of matching overhead. Overall, TTF offers a practical, efficient solution for video understanding. The code is available at \href{https://github.com/Cominder/ttf}{https://github.com/Cominder/ttf}