V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models

TL;DR

V-CAST is a novel, training-free token pruning method for VideoLLMs that improves long-context video inference efficiency by preserving spatio-temporal information and reducing memory and latency.

Contribution

It introduces a curvature-guided temporal allocation and dual-anchor spatial selection mechanism for effective token compression without training.

Findings

Achieves 98.6% of original performance on VideoLLMs.

Reduces peak memory and latency to approximately 86% of baseline.

Outperforms existing methods by 1.1% on average.

Abstract

Video large language models (VideoLLMs) show strong capability in video understanding, yet long-context inference is still dominated by massive redundant visual tokens in the prefill stage. We revisit token compression for VideoLLMs under a tight budget and identify a key bottleneck, namely insufficient spatio-temporal information coverage. Existing methods often introduce discontinuous coverage through coarse per-frame allocation or scene segmentation, and token merging can further misalign spatio-temporal coordinates under MRoPE-style discrete (t,h,w) bindings. To address these issues, we propose V-CAST (Video Curvature-Aware Spatio-Temporal Pruning), a training-free, plug-and-play pruning policy for long-context video inference. V-CAST casts token compression as a trajectory approximation problem and introduces a curvature-guided temporal allocation module that routes per-frame token budgets to semantic turns and event boundaries. It further adopts a dual-anchor spatial selection mechanism that preserves high-entropy visual evidence without attention intervention, while keeping retained tokens at their original coordinates to maintain positional alignment. Extensive experiments across multiple VideoLLMs of different architectures and scales demonstrate that V-CAST achieves 98.6% of the original performance, outperforms the second-best method by +1.1% on average, and reduces peak memory and total latency to 86.7% and 86.4% of vanilla Qwen3-VL-8B-Instruct.