When Does Pruning Benefit Vision Representations?

TL;DR

This paper explores how pruning affects vision models' interpretability, object discovery, and alignment with human perception, revealing that benefits depend on architecture and sparsity levels.

Contribution

It provides a comprehensive analysis of pruning's impact on interpretability, unsupervised object discovery, and human alignment across different vision architectures.

Findings

Sparse models can improve interpretability and human alignment.

Optimal sparsity levels vary with architecture and size.

Pruning can enhance unsupervised object discovery by removing redundancy.

Abstract

Pruning is widely used to reduce the complexity of deep learning models, but its effects on interpretability and representation learning remain poorly understood. This paper investigates how pruning influences vision models across three key dimensions: (i) interpretability, (ii) unsupervised object discovery, and (iii) alignment with human perception. We first analyze different vision network architectures to examine how varying sparsity levels affect feature attribution interpretability methods. Additionally, we explore whether pruning promotes more succinct and structured representations, potentially improving unsupervised object discovery by discarding redundant information while preserving essential features. Finally, we assess whether pruning enhances the alignment between model representations and human perception, investigating whether sparser models focus on more discriminative features similarly to humans. Our findings also reveal the presence of sweet spots, where sparse models exhibit higher interpretability, downstream generalization and human alignment. However, these spots highly depend on the network architectures and their size in terms of trainable parameters. Our results suggest a complex interplay between these three dimensions, highlighting the importance of investigating when and how pruning benefits vision representations.