A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs

TL;DR

This paper introduces SGL, a training-free method that uses small vision-language models to guide token pruning in large models, significantly accelerating inference while maintaining high accuracy.

Contribution

The paper proposes a novel, training-free approach leveraging small VLM attention maps to efficiently prune tokens in large VLMs, with an early exit mechanism for improved speed and accuracy.

Findings

Achieves up to 91% token pruning with minimal performance loss

Global attention maps from small VLMs closely resemble those of large VLMs

Method outperforms existing pruning techniques across 11 benchmarks

Abstract

Vision-language models (VLMs) have shown remarkable success across various multi-modal tasks, yet large VLMs encounter significant efficiency challenges due to processing numerous visual tokens. A promising approach to accelerating large VLM inference is using partial information, such as attention maps from specific layers, to assess token importance and prune less essential tokens. However, our study reveals three key insights: (i) Partial attention information is insufficient for accurately identifying critical visual tokens, resulting in suboptimal performance, especially at low token retention ratios; (ii) Global attention information, such as the attention map aggregated across all layers, more effectively preserves essential tokens and maintains comparable performance under aggressive pruning. However, the attention maps from all layers requires a full inference pass, which increases computational load and is therefore impractical in existing methods; and (iii) The global attention map aggregated from a small VLM closely resembles that of a large VLM, suggesting an efficient alternative. Based on these findings, we introduce a \textbf{training-free} method, \underline{\textbf{S}}mall VLM \underline{\textbf{G}}uidance for accelerating \underline{\textbf{L}}arge VLMs (\textbf{SGL}). Specifically, we employ the attention map aggregated from a small VLM to guide visual token pruning in a large VLM. Additionally, an early exiting mechanism is developed to fully use the small VLM's predictions, dynamically invoking the larger VLM only when necessary, yielding a superior trade-off between accuracy and computation. Extensive evaluations across 11 benchmarks demonstrate the effectiveness and generalizability of SGL, achieving up to 91\% pruning ratio for visual tokens while retaining competitive performance.