Towards Fine-Grained Interpretability: Counterfactual Explanations for Misclassification with Saliency Partition

TL;DR

This paper introduces a novel fine-grained counterfactual explanation framework that enhances interpretability of misclassification in visual models by providing object- and part-level insights, surpassing existing methods.

Contribution

It proposes a non-generative, saliency partition-based approach for detailed, region-specific interpretability of model misclassifications at object and part levels.

Findings

Outperforms existing fine-grained explanation methods.

Provides more intuitive and detailed local feature insights.

Effectively isolates region-specific feature relevance.

Abstract

Attribution-based explanation techniques capture key patterns to enhance visual interpretability; however, these patterns often lack the granularity needed for insight in fine-grained tasks, particularly in cases of model misclassification, where explanations may be insufficiently detailed. To address this limitation, we propose a fine-grained counterfactual explanation framework that generates both object-level and part-level interpretability, addressing two fundamental questions: (1) which fine-grained features contribute to model misclassification, and (2) where dominant local features influence counterfactual adjustments. Our approach yields explainable counterfactuals in a non-generative manner by quantifying similarity and weighting component contributions within regions of interest between correctly classified and misclassified samples. Furthermore, we introduce a saliency partition module grounded in Shapley value contributions, isolating features with region-specific relevance. Extensive experiments demonstrate the superiority of our approach in capturing more granular, intuitively meaningful regions, surpassing fine-grained methods.