Towards Efficient VLMs: Information-Theoretic Driven Compression via Adaptive Structural Pruning

TL;DR

This paper introduces InfoPrune, an information-theoretic framework for adaptively compressing vision-language models by pruning attention heads and low-rank approximations, achieving significant efficiency gains with minimal performance loss.

Contribution

The work presents a novel, theoretically grounded method for VLM compression using the Information Bottleneck principle, entropy-based metrics, and adaptive pruning schemes.

Findings

Achieves up to 3.2x FLOP reduction and 1.8x acceleration.

Maintains performance with negligible degradation on VQAv2, TextVQA, and GQA.

Provides a unified, information-theoretic criterion for structural sparsity and efficiency.

Abstract

Recent advances in vision-language models (VLMs) have shown remarkable performance across multimodal tasks, yet their ever-growing scale poses severe challenges for deployment and efficiency. Existing compression methods often rely on heuristic importance metrics or empirical pruning rules, lacking theoretical guarantees about information preservation. In this work, we propose InfoPrune, an information-theoretic framework for adaptive structural compression of VLMs. Grounded in the Information Bottleneck principle, we formulate pruning as a trade-off between retaining task-relevant semantics and discarding redundant dependencies. To quantify the contribution of each attention head, we introduce an entropy-based effective rank (eRank) and employ the Kolmogorov--Smirnov (KS) distance to measure the divergence between original and compressed structures. This yields a unified criterion that jointly considers structural sparsity and informational efficiency. Building on this foundation, we further design two complementary schemes: (1) a training-based head pruning guided by the proposed information loss objective, and (2) a training-free FFN compression via adaptive low-rank approximation. Extensive experiments on VQAv2, TextVQA, and GQA demonstrate that InfoPrune achieves up to 3.2x FLOP reduction and 1.8x acceleration with negligible performance degradation, establishing a theoretically grounded and practically effective step toward efficient multimodal large models.