IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruning

TL;DR

This paper reveals that LVLMs implicitly establish visual coordinate systems via Rotary Position Embeddings, and introduces IVC-Prune, a training-free token pruning method that preserves spatial reasoning tokens, reducing computational cost while maintaining performance.

Contribution

The paper uncovers the implicit visual coordinate system in LVLMs and proposes IVC-Prune, a novel token pruning strategy that retains spatial reasoning tokens without additional training.

Findings

Reduces visual tokens by ~50% while maintaining ≥99% performance.

Identifies IVC tokens through mathematical analysis of RoPE.

Achieves performance improvements on several benchmarks.

Abstract

Large Vision-Language Models (LVLMs) achieve impressive performance across multiple tasks. A significant challenge, however, is their prohibitive inference cost when processing high-resolution visual inputs. While visual token pruning has emerged as a promising solution, existing methods that primarily focus on semantic relevance often discard tokens that are crucial for spatial reasoning. We address this gap through a novel insight into \emph{how LVLMs process spatial reasoning}. Specifically, we reveal that LVLMs implicitly establish visual coordinate systems through Rotary Position Embeddings (RoPE), where specific token positions serve as \textbf{implicit visual coordinates} (IVC tokens) that are essential for spatial reasoning. Based on this insight, we propose \textbf{IVC-Prune}, a training-free, prompt-aware pruning strategy that retains both IVC tokens and semantically relevant foreground tokens. IVC tokens are identified by theoretically analyzing the mathematical properties of RoPE, targeting positions at which its rotation matrices approximate identity matrix or the $90^\circ$ rotation matrix. Foreground tokens are identified through a robust two-stage process: semantic seed discovery followed by contextual refinement via value-vector similarity. Extensive evaluations across four representative LVLMs and twenty diverse benchmarks show that IVC-Prune reduces visual tokens by approximately 50\% while maintaining $\geq$ 99\% of the original performance and even achieving improvements on several benchmarks. Source codes are available at https://github.com/FireRedTeam/IVC-Prune.