Energy-Regularized Spatial Masking: A Novel Approach to Enhancing Robustness and Interpretability in Vision Models

TL;DR

Energy-Regularized Spatial Masking (ERSM) introduces a differentiable energy-based feature selection method that enhances robustness and interpretability in vision models by autonomously discovering optimal spatial masks.

Contribution

ERSM reformulates feature selection as a differentiable energy minimization, enabling autonomous discovery of spatial masks that improve robustness and interpretability without fixed sparsity constraints.

Findings

ERSM produces emergent sparsity and interpretable masks.

ERSM improves robustness to structured occlusion.

Energy ranking outperforms magnitude-based pruning in robustness tests.

Abstract

Deep convolutional neural networks achieve remarkable performance by exhaustively processing dense spatial feature maps, yet this brute-force strategy introduces significant computational redundancy and encourages reliance on spurious background correlations. As a result, modern vision models remain brittle and difficult to interpret. We propose Energy-Regularized Spatial Masking (ERSM), a novel framework that reformulates feature selection as a differentiable energy minimization problem. By embedding a lightweight Energy-Mask Layer inside standard convolutional backbones, each visual token is assigned a scalar energy composed of two competing forces: an intrinsic Unary importance cost and a Pairwise spatial coherence penalty. Unlike prior pruning methods that enforce rigid sparsity budgets or rely on heuristic importance scores, ERSM allows the network to autonomously discover an optimal information-density equilibrium tailored to each input. We validate ERSM on convolutional architectures and demonstrate that it produces emergent sparsity, improved robustness to structured occlusion, and highly interpretable spatial masks, while preserving classification accuracy. Furthermore, we show that the learned energy ranking significantly outperforms magnitude-based pruning in deletion-based robustness tests, revealing ERSM as an intrinsic denoising mechanism that isolates semantic object regions without pixel-level supervision.