AIM: Amending Inherent Interpretability via Self-Supervised Masking

TL;DR

AIM is a method that enhances neural network interpretability and accuracy by self-supervised masking of features, promoting genuine feature use without extra annotations, validated across diverse datasets.

Contribution

The paper introduces AIM, a novel self-supervised masking technique that improves both interpretability and performance of neural networks without additional labels.

Findings

AIM improves interpretability as measured by EPG scores.

AIM enhances accuracy across multiple datasets.

AIM promotes genuine feature utilization leading to better generalization.

Abstract

It has been observed that deep neural networks (DNNs) often use both genuine as well as spurious features. In this work, we propose "Amending Inherent Interpretability via Self-Supervised Masking" (AIM), a simple yet interestingly effective method that promotes the network's utilization of genuine features over spurious alternatives without requiring additional annotations. In particular, AIM uses features at multiple encoding stages to guide a self-supervised, sample-specific feature-masking process. As a result, AIM enables the training of well-performing and inherently interpretable models that faithfully summarize the decision process. We validate AIM across a diverse range of challenging datasets that test both out-of-distribution generalization and fine-grained visual understanding. These include general-purpose classification benchmarks such as ImageNet100, HardImageNet, and ImageWoof, as well as fine-grained classification datasets such as Waterbirds, TravelingBirds, and CUB-200. AIM demonstrates significant dual benefits: interpretability improvements, as measured by the Energy Pointing Game (EPG) score, and accuracy gains over strong baselines. These consistent gains across domains and architectures provide compelling evidence that AIM promotes the use of genuine and meaningful features that directly contribute to improved generalization and human-aligned interpretability.