Mechanistically Interpreting Compression in Vision-Language Models

TL;DR

This paper investigates how different compression techniques like pruning and quantization affect the internal mechanisms and safety behaviors of vision-language models, revealing distinct impacts on model internals and safety performance.

Contribution

It introduces a causal circuit analysis framework to understand compression effects and proposes VLMSafe-420, a benchmark for safety evaluation under compression.

Findings

Pruning preserves circuit structure but alters features.

Quantization modifies circuits while maintaining feature alignment.

Compression impacts safety behaviors, notably reducing genuine refusal responses.

Abstract

Compressed vision-language models (VLMs) are widely used to reduce memory and compute costs, making them a suitable choice for real-world deployment. However, compressing these models raises concerns about whether internal computations and safety behaviors are preserved. In this work, we use causal circuit analysis and crosscoder-based feature comparisons to examine how pruning and quantization fundamentally change the internals across representative VLMs. We observe that pruning generally keeps circuit structure intact but rotates and attenuates internal features, while quantization modifies the circuits at a higher level yet leaves the surviving features better aligned. Leveraging this insight, we also introduce VLMSafe-420, a novel benchmark that pairs harmful inputs with matched benign counterfactuals across various safety categories. Our findings show that pruning causes a sharp drop in genuine refusal behavior, suggesting that the choice of compression has safety implications.