Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Models

TL;DR

This paper introduces Turbo, a plug-in module that accelerates vision-language models by pruning tokens based on their information content, effectively reducing computation costs while maintaining performance.

Contribution

It pioneers a data-centric approach to model acceleration by designing an information degree-guided token pruning method applicable across various VLMs.

Findings

Significant speed-up in VLMs with minimal performance loss

Compatible with multiple VLM architectures and tasks

Simple plug-in design requiring no retraining

Abstract

Vision-Language Large Models (VLMs) have become primary backbone of AI, due to the impressive performance. However, their expensive computation costs, i.e., throughput and delay, impede potentials in real-world scenarios. To achieve acceleration for VLMs, most existing methods focus on the model perspective: pruning, distillation, quantification, but completely overlook the data-perspective redundancy. To fill the overlook, this paper pioneers the severity of data redundancy, and designs one plug-and-play Turbo module guided by information degree to prune inefficient tokens from visual or textual data. In pursuit of efficiency-performance trade-offs, information degree takes two key factors into consideration: mutual redundancy and semantic value. Concretely, the former evaluates the data duplication between sequential tokens; while the latter evaluates each token by its contribution to the overall semantics. As a result, tokens with high information degree carry less redundancy and stronger semantics. For VLMs' calculation, Turbo works as a user-friendly plug-in that sorts data referring to information degree, utilizing only top-level ones to save costs. Its advantages are multifaceted, e.g., being generally compatible to various VLMs across understanding and generation, simple use without retraining and trivial engineering efforts. On multiple public VLMs benchmarks, we conduct extensive experiments to reveal the gratifying acceleration of Turbo, under negligible performance drop.