Understanding Pruning Regimes in Vision-Language Models Through Domain-Aware Layer Selection

TL;DR

This paper investigates how pruning specific layers in vision-language models affects their performance across math and general tasks, revealing a three-regime structure and proposing domain-aware pruning strategies.

Contribution

It introduces domain-aware activation similarity for layer pruning, providing insights into depth redundancy and a practical approach to reduce model size without losing capabilities.

Findings

Performance sensitivity varies with pruning budget

Three distinct regimes of pruning effects identified

Domain-aware rankings improve stability in pruning

Abstract

Transformer-based vision-language models (VLMs) contain substantial depth redundancy, yet the effect of removing specific decoder layers remains poorly understood, especially for domains that require tight coupling between perception and multi-step reasoning. We study structured decoder layer pruning through the lens of domain-aware activation similarity, measuring how strongly each layer transforms representations for math versus non-math inputs. This yields simple math-aware, non-math-aware, and mixed ranking criteria that identify layers whose input-output activations change least within a target domain. Across two state-of-the-art VLMs and a broad suite of math and general multimodal benchmarks, we uncover a consistent three-regime structure: at low pruning budgets, performance is highly sensitive to which layers are removed; at moderate budgets, methods converge as structural damage accumulates; and at high budgets, structural continuity dominates, favoring spacing-aware strategies. Our domain-aware rankings achieve the strongest stability in the ranking-sensitive regime, while matching or exceeding structure-aware baselines at larger budgets. These results provide a clearer picture of how depth contributes to domain-specific behavior in VLMs and offer a practical, interpretable approach to reducing model depth without sacrificing essential mathematical or general vision-language capabilities.