TiFRe: Text-guided Video Frame Reduction for Efficient Video Multi-modal Large Language Models

TL;DR

TiFRe is a novel framework that intelligently reduces video frames for multi-modal LLMs by selecting and merging frames based on user input, significantly lowering computation while maintaining or improving task performance.

Contribution

The paper introduces TiFRe, a text-guided frame reduction method that preserves essential video information through semantic-aware sampling and merging, enhancing efficiency in video-language models.

Findings

Reduces computational costs significantly.

Improves performance on video-language tasks.

Effectively preserves video semantics during reduction.

Abstract

With the rapid development of Large Language Models (LLMs), Video Multi-Modal Large Language Models (Video MLLMs) have achieved remarkable performance in video-language tasks such as video understanding and question answering. However, Video MLLMs face high computational costs, particularly in processing numerous video frames as input, which leads to significant attention computation overhead. A straightforward approach to reduce computational costs is to decrease the number of input video frames. However, simply selecting key frames at a fixed frame rate (FPS) often overlooks valuable information in non-key frames, resulting in notable performance degradation. To address this, we propose Text-guided Video Frame Reduction (TiFRe), a framework that reduces input frames while preserving essential video information. TiFRe uses a Text-guided Frame Sampling (TFS) strategy to select key frames based on user input, which is processed by an LLM to generate a CLIP-style prompt. Pre-trained CLIP encoders calculate the semantic similarity between the prompt and each frame, selecting the most relevant frames as key frames. To preserve video semantics, TiFRe employs a Frame Matching and Merging (FMM) mechanism, which integrates non-key frame information into the selected key frames, minimizing information loss. Experiments show that TiFRe effectively reduces computational costs while improving performance on video-language tasks.