Short-LVLM: Compressing and Accelerating Large Vision-Language Models by Pruning Redundant Layers

TL;DR

This paper introduces Short-LVLM, a novel framework for compressing large vision-language models by pruning redundant layers, focusing on preserving important tokens and reducing feature gaps, resulting in efficient models without retraining.

Contribution

The paper demonstrates the ineffectiveness of NLP layer pruning methods on LVLMs and proposes a new, training-free, model-agnostic framework that improves efficiency while maintaining performance.

Findings

Short-LVLM achieves better performance-efficiency trade-offs.

It is training-free and highly compatible with existing models.

The method effectively preserves important vision-language tokens.

Abstract

Although large vision-language models (LVLMs) have demonstrated impressive capabilities in multi-modal understanding and reasoning, their practical applications are still limited by massive model parameters and high computational costs. Recent efforts from natural language processing (NLP) have shown the effectiveness of layer pruning, offering a plausible training-free compression solution. However, due to the modality divergence between vision and language, it is unclear whether these NLP techniques are still effective in LVLMs. In this paper, we empirically prove that directly applying these layer pruning methods to LVLMs is ineffective. Through extensive experiments, we find that non-essential vision-language (VL) tokens and inter-layer feature gaps pose critical challenges to pruning layers in LVLMs. Based on these insights, we propose a novel framework Short-LVLM (SVL) that can utilize important VL tokens and mitigate the layer-wise feature gaps. Notably, Short-LVLM not only achieves a superior trade-off between performance and efficiency but also exhibits several potential advantages, i.e., training-free, model-agnostic, and highly compatible. The code for this work is publicly available at https://github.com/ASGO-MM/Short-LVLM.