EVTP-IVS: Effective Visual Token Pruning For Unifying Instruction Visual Segmentation In Multi-Modal Large Language Models

TL;DR

This paper introduces EVTP-IV, a visual token pruning method for instruction-based visual segmentation in multimodal large language models, significantly speeding up inference while maintaining accuracy.

Contribution

It proposes a novel token pruning technique that improves inference efficiency in IVS tasks by selecting a spatially representative token subset, supported by an information-theoretic analysis.

Findings

Achieves up to 5X speed-up on video IVS tasks

Maintains comparable accuracy with only 20% of tokens

Outperforms existing pruning methods across benchmarks

Abstract

Instructed Visual Segmentation (IVS) tasks require segmenting objects in images or videos based on natural language instructions. While recent multimodal large language models (MLLMs) have achieved strong performance on IVS, their inference cost remains a major bottleneck, particularly in video. We empirically analyze visual token sampling in MLLMs and observe a strong correlation between subset token coverage and segmentation performance. This motivates our design of a simple and effective token pruning method that selects a compact yet spatially representative subset of tokens to accelerate inference. In this paper, we introduce a novel visual token pruning method for IVS, called EVTP-IV, which builds upon the k-center by integrating spatial information to ensure better coverage. We further provide an information-theoretic analysis to support our design. Experiments on standard IVS benchmarks show that our method achieves up to 5X speed-up on video tasks and 3.5X on image tasks, while maintaining comparable accuracy using only 20% of the tokens. Our method also consistently outperforms state-of-the-art pruning baselines under varying pruning ratios.