KTV: Keyframes and Key Tokens Selection for Efficient Training-Free Video LLMs

TL;DR

KTV introduces a two-stage method for efficient training-free video understanding by selecting keyframes and tokens, reducing redundancy and computational load while improving accuracy on video question-answering tasks.

Contribution

The paper proposes KTV, a novel framework that combines question-agnostic keyframe clustering with token importance-based pruning to enhance training-free video comprehension.

Findings

Outperforms state-of-the-art training-free methods on Multiple-Choice VideoQA.

Uses only 504 visual tokens for a 60-minute video, significantly reducing computational cost.

Achieves 44.8% accuracy on the MLVU-Test benchmark.

Abstract

Training-free video understanding leverages the strong image comprehension capabilities of pre-trained vision language models (VLMs) by treating a video as a sequence of static frames, thus obviating the need for costly video-specific training. However, this paradigm often suffers from severe visual redundancy and high computational overhead, especially when processing long videos. Crucially, existing keyframe selection strategies, especially those based on CLIP similarity, are prone to biases and may inadvertently overlook critical frames, resulting in suboptimal video comprehension. To address these significant challenges, we propose \textbf{KTV}, a novel two-stage framework for efficient and effective training-free video understanding. In the first stage, KTV performs question-agnostic keyframe selection by clustering frame-level visual features, yielding a compact, diverse, and representative subset of frames that mitigates temporal redundancy. In the second stage, KTV applies key visual token selection, pruning redundant or less informative tokens from each selected keyframe based on token importance and redundancy, which significantly reduces the number of tokens fed into the LLM. Extensive experiments on the Multiple-Choice VideoQA task demonstrate that KTV outperforms state-of-the-art training-free baselines while using significantly fewer visual tokens, \emph{e.g.}, only 504 visual tokens for a 60-min video with 10800 frames, achieving $44.8\%$ accuracy on the MLVU-Test benchmark. In particular, KTV also exceeds several training-based approaches on certain benchmarks.