CAPA: Contribution-Aware Pruning and FFN Approximation for Efficient Large Vision-Language Models

TL;DR

CAPA introduces a novel framework for pruning visual tokens and approximating FFN computations in large vision-language models, leading to improved efficiency and robustness without significant performance loss.

Contribution

It proposes Attention Contribution as a better criterion for token importance and introduces a dual strategy for token pruning and FFN approximation in vision-language models.

Findings

CAPA achieves significant efficiency gains with minimal performance loss.

Attention Contribution outperforms traditional attention scores in token importance estimation.

Redundancy in FFNs allows for effective linear approximations.

Abstract

Efficient inference in Large Vision-Language Models is constrained by the high cost of processing thousands of visual tokens, yet it remains unclear which tokens and computations can be safely removed. While attention scores are commonly used to estimate visual token importance, they are an imperfect proxy for actual contribution. We show that Attention Contribution, which weights attention probabilities by value vector magnitude, provides a more accurate criterion for visual token selection. Our empirical analysis reveals that visual attention sinks are functionally heterogeneous, comprising Probability Dumps with low contribution that can be safely pruned, and Structural Anchors with high contribution essential for maintaining model performance. Further, we identify substantial redundancy in Feed-Forward Networks (FFNs) associated with visual tokens, particularly in intermediate layers where image tokens exhibit linear behavior. Based on our findings, we introduce CAPA (Contribution-Aware Pruning and FFN Approximation), a dual-strategy framework that prunes visual tokens using attention contribution at critical functional transitions and reduces FFN computation through efficient linear approximations. Experiments on various benchmarks across baselines show that CAPA achieves competent efficiency--performance trade-offs with improved robustness.