Causal Explanations of Image Misclassifications

TL;DR

This paper investigates the causes of image misclassifications in CNNs trained on CIFAR-10, identifying morphological similarity and non-essential information interference as key factors, and proposes a pixel erasure method to reduce errors.

Contribution

It introduces a novel approach to analyze and distinguish causes of misclassification and proposes a targeted pixel erasure technique to mitigate non-essential information interference.

Findings

Identified two main causes of misclassification: morphological similarity and non-essential information interference.

Proposed a pixel erasure method that reduces misclassification caused by non-essential information.

Demonstrated that the cause of misclassification can be verified and directly addressed through saliency-based pixel removal.

Abstract

The causal explanation of image misclassifications is an understudied niche, which can potentially provide valuable insights in model interpretability and increase prediction accuracy. This study trains CIFAR-10 on six modern CNN architectures, including VGG16, ResNet50, GoogLeNet, DenseNet161, MobileNet V2, and Inception V3, and explores the misclassification patterns using conditional confusion matrices and misclassification networks. Two causes are identified and qualitatively distinguished: morphological similarity and non-essential information interference. The former cause is not model dependent, whereas the latter is inconsistent across all six models. To reduce the misclassifications caused by non-essential information interference, this study erases the pixels within the bonding boxes anchored at the top 5% pixels of the saliency map. This method first verifies the cause; then by directly modifying the cause it reduces the misclassification. Future studies will focus on quantitatively differentiating the two causes of misclassifications, generalizing the anchor-box based inference modification method to reduce misclassification, exploring the interactions of the two causes in misclassifications.