PruneVid: Visual Token Pruning for Efficient Video Large Language Models

TL;DR

PruneVid is a training-free visual token pruning technique that significantly reduces video data redundancy, enabling large language models to process videos more efficiently without substantial performance loss.

Contribution

The paper introduces a novel, training-free token pruning method for large video models that merges spatial-temporal tokens and selectively prunes relevant features using LLM reasoning.

Findings

PruneVid can prune over 80% of tokens while maintaining performance.

The method outperforms existing pruning techniques in efficiency and effectiveness.

Validated across multiple video benchmarks.

Abstract

In this paper, we introduce PruneVid, a visual token pruning method designed to enhance the efficiency of multi-modal video understanding. Large Language Models (LLMs) have shown promising performance in video tasks due to their extended capabilities in comprehending visual modalities. However, the substantial redundancy in video data presents significant computational challenges for LLMs. To address this issue, we introduce a training-free method that 1) minimizes video redundancy by merging spatial-temporal tokens, and 2) leverages LLMs' reasoning capabilities to selectively prune visual features relevant to question tokens, enhancing model efficiency. We validate our method across multiple video benchmarks, which demonstrate that PruneVid can prune over 80% of tokens while maintaining competitive performance combined with different model networks. This highlights its superior effectiveness and efficiency compared to existing pruning methods. Code: https://github.com/Visual-AI/PruneVid.