SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass

TL;DR

SwiftVLM introduces a layer-wise token bypass approach for vision-language models, enabling more accurate and efficient visual token pruning without early irreversible decisions, thus improving performance on fine-grained tasks.

Contribution

The paper proposes a novel bypass pruning paradigm and SwiftVLM method that preserve and re-evaluate visual tokens across layers, enhancing pruning flexibility and accuracy.

Findings

Outperforms existing pruning strategies across multiple benchmarks.

Achieves better accuracy-efficiency trade-offs.

Demonstrates more faithful visual token selection behavior.

Abstract

Visual token pruning is a promising approach for reducing the computational cost of vision-language models (VLMs), and existing methods often rely on early pruning decisions to improve efficiency. While effective on coarse-grained reasoning tasks, they suffer from significant performance degradation on tasks requiring fine-grained visual details. Through layer-wise analysis, we reveal substantial discrepancies in visual token importance across layers, showing that tokens deemed unimportant at shallow layers can later become highly relevant for text-conditioned reasoning. To avoid irreversible critical information loss caused by premature pruning, we introduce a new pruning paradigm, termed bypass, which preserves unselected visual tokens and forwards them to subsequent pruning stages for re-evaluation. Building on this paradigm, we propose SwiftVLM, a simple and training-free method that performs pruning at model-specific layers with strong visual token selection capability, while enabling independent pruning decisions across layers. Experiments across multiple VLMs and benchmarks demonstrate that SwiftVLM consistently outperforms existing pruning strategies, achieving superior accuracy-efficiency trade-offs and more faithful visual token selection behavior.