SIC: Similarity-Based Interpretable Image Classification with Neural Networks

TL;DR

SIC is an interpretable neural network that uses case-based reasoning with support vectors and similarity scores to provide both local and global explanations, achieving competitive accuracy in image classification tasks.

Contribution

We propose SIC, a novel neural network architecture that inherently offers interpretable, case-based explanations while maintaining high classification accuracy.

Findings

SIC achieves competitive accuracy on multiple image classification benchmarks.

SIC provides coherent pixel-level and global explanations verified through practical evaluation.

Theoretical analysis confirms SIC's explanations meet established axioms for interpretability.

Abstract

The deployment of deep learning models in critical domains necessitates a balance between high accuracy and interpretability. We introduce SIC, an inherently interpretable neural network that provides local and global explanations of its decision-making process. Leveraging the concept of case-based reasoning, SIC extracts class-representative support vectors from training images, ensuring they capture relevant features while suppressing irrelevant ones. Classification decisions are made by calculating and aggregating similarity scores between these support vectors and the input's latent feature vector. We employ B-Cos transformations, which align model weights with inputs, to yield coherent pixel-level explanations in addition to global explanations of case-based reasoning. We evaluate SIC on three tasks: fine-grained classification on Stanford Dogs and FunnyBirds, multi-label classification on Pascal VOC, and pathology detection on the RSNA dataset. Results indicate that SIC not only achieves competitive accuracy compared to state-of-the-art black-box and inherently interpretable models but also offers insightful explanations verified through practical evaluation on the FunnyBirds benchmark. Our theoretical analysis proves that these explanations fulfill established axioms for explanations. Our findings underscore SIC's potential for applications where understanding model decisions is as critical as the decisions themselves.