Sparse but not Simpler: A Multi-Level Interpretability Analysis of Vision Transformers

TL;DR

This paper systematically evaluates whether structural sparsity in Vision Transformers improves interpretability, finding that pruning reduces circuit complexity but does not enhance interpretability at multiple levels.

Contribution

Introduces IMPACT, a comprehensive multi-level framework for evaluating interpretability in vision models, and provides empirical evidence that sparsity alone does not improve interpretability.

Findings

Sparse models have fewer circuit edges but similar or more active nodes.

Pruning redistributes computation rather than simplifying functions.

No significant improvements in neuron selectivity or attribution faithfulness.

Abstract

Sparse neural networks are often hypothesized to be more interpretable than dense models, motivated by findings that weight sparsity can produce compact circuits in language models. However, it remains unclear whether structural sparsity itself leads to improved semantic interpretability. In this work, we systematically evaluate the relationship between weight sparsity and interpretability in Vision Transformers using DeiT-III B/16 models pruned with Wanda. To assess interpretability comprehensively, we introduce \textbf{IMPACT}, a multi-level framework that evaluates interpretability across four complementary levels: neurons, layer representations, task circuits, and model-level attribution. Layer representations are analyzed using BatchTopK sparse autoencoders, circuits are extracted via learnable node masking, and explanations are evaluated with transformer attribution using insertion and deletion metrics. Our results reveal a clear structural effect but limited interpretability gains. Sparse models produce circuits with approximately $2.5\times$ fewer edges than dense models, yet the fraction of active nodes remains similar or higher, indicating that pruning redistributes computation rather than isolating simpler functional modules. Consistent with this observation, sparse models show no systematic improvements in neuron-level selectivity, SAE feature interpretability, or attribution faithfulness. These findings suggest that structural sparsity alone does not reliably yield more interpretable vision models, highlighting the importance of evaluation frameworks that assess interpretability beyond circuit compactness.