Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Models

TL;DR

This paper introduces Turbo, a plug-in module that accelerates vision-language models by pruning tokens based on their information content, effectively reducing computation costs while maintaining performance.

Contribution

The paper pioneers a data-centric approach to model acceleration by designing an information degree-guided token pruning method applicable across various VLMs.

Findings

Turbo achieves significant speed-up with negligible performance loss.

The method is compatible with multiple VLM architectures.

It requires no re-training or complex engineering.

Abstract

Vision-Language Large Models (VLMs) recently become primary backbone of AI, due to the impressive performance. However, their expensive computation costs, i.e., throughput and delay, impede potentials in the real-world scenarios. To achieve acceleration for VLMs, most existing methods focus on the model perspective: pruning, distillation, quantization, but completely overlook the data-perspective redundancy. To fill the overlook, this paper pioneers the severity of data redundancy, and designs one plug-and-play Turbo module guided by information degree to prune inefficient tokens from visual or textual data. In pursuit of efficiency-performance trade-offs, information degree takes two crucial factors into consideration: mutual redundancy and semantic value. Concretely, the former evaluates data duplication between sequential tokens; while the latter evaluates each token by its contribution to the overall semantics. As a result, tokens with high information degree carry less redundancy and stronger semantics. For VLMs' calculation, Turbo works as a user-friendly plug-in that sorts data referring to information degree, utilizing only top-level ones to save costs. Its advantages are multifaceted, e.g., being generally compatible to various VLMs across understanding and generation, simple use without re-training and trivial engineering efforts. On multiple VLMs benchmarks, we fully experiment to demonstrate the good acceleration of Turbo, under negligible performance drop.