Spatial Information Bottleneck for Interpretable Visual Recognition

TL;DR

This paper introduces Spatial Information Bottleneck (S-IB), a method that improves neural network interpretability and robustness by spatially disentangling class-relevant information from background noise, leading to better explanations and accuracy.

Contribution

The paper presents a novel information-theoretic framework and a spatial disentanglement method (S-IB) that enhances interpretability and robustness of neural networks.

Findings

Improved visualization quality across multiple explanation methods.

Enhanced foreground focus and background suppression in explanations.

Consistent accuracy improvements on five benchmarks.

Abstract

Deep neural networks typically learn spatially entangled representations that conflate discriminative foreground features with spurious background correlations, thereby undermining model interpretability and robustness. We propose a novel understanding framework for gradient-based attribution from an information-theoretic perspective. We prove that, under mild conditions, the Vector-Jacobian Products (VJP) computed during backpropagation form minimal sufficient statistics of input features with respect to class labels. Motivated by this finding, we propose an encoding-decoding perspective : forward propagation encodes inputs into class space, while VJP in backpropagation decodes this encoding back to feature space. Therefore, we propose Spatial Information Bottleneck (S-IB) to spatially disentangle information flow. By maximizing mutual information between foreground VJP and inputs while minimizing mutual information in background regions, S-IB encourages networks to encode information only in class-relevant spatial regions. Since post-hoc explanation methods fundamentally derive from VJP computations, directly optimizing VJP's spatial structure during training improves visualization quality across diverse explanation paradigms. Experiments on five benchmarks demonstrate universal improvements across six explanation methods, achieving better foreground concentration and background suppression without method-specific tuning, alongside consistent classification accuracy gains.