Enhancing Visual Token Representations for Video Large Language Models via Training-Free Spatial-Temporal Pooling and Gridding

TL;DR

This paper introduces ST-GridPool, a training-free method that enhances visual token representations in Video LLMs by capturing multi-scale spatiotemporal interactions and preserving semantic-rich regions, leading to improved performance.

Contribution

The paper proposes a novel training-free approach combining Pyramid Temporal Gridding and Norm-based Spatial Pooling to improve visual token encoding in Video LLMs.

Findings

Consistently improves Video LLM performance across benchmarks.

Does not require retraining, saving computational resources.

Effectively captures multi-grained spatiotemporal interactions.

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have significantly advanced video understanding tasks, yet challenges remain in efficiently compressing visual tokens while preserving spatiotemporal interactions. Existing methods, such as LLaVA family, utilize simplistic pooling or interpolation techniques that overlook the intricate dynamics of visual tokens. To bridge this gap, we propose ST-GridPool, a novel training-free visual token enhancement method designed specifically for Video LLMs. Our approach integrates Pyramid Temporal Gridding (PTG), which captures multi-grained spatiotemporal interactions through hierarchical temporal gridding, and Norm-based Spatial Pooling (NSP), which preserves high-information visual regions by leveraging the correlation between token norms and semantic richness. Extensive experiments on various benchmarks demonstrate that ST-GridPool consistently enhances performance of Video LLMs without requiring costly retraining. Our method offers an efficient and plug-and-play solution for improving visual token representations. Our code is available in https://github.com/bingjunluo/ST-GridPool.