Game-Theoretic Understanding of Misclassification

TL;DR

This paper uses game theory to analyze different types of image misclassification, revealing how adversarial and corrupted images differ in pixel interactions and comparing CNNs with Vision Transformers.

Contribution

It introduces a novel game-theoretic framework to analyze misclassification types and applies it to both CNNs and Vision Transformers, uncovering distinct interaction patterns.

Findings

Adversarial images have higher high-order interactions than clean images.

Corrupted images exhibit lower low-order interactions than clean images.

Vision Transformers show different interaction distributions compared to CNNs.

Abstract

This paper analyzes various types of image misclassification from a game-theoretic view. Particularly, we consider the misclassification of clean, adversarial, and corrupted images and characterize it through the distribution of multi-order interactions. We discover that the distribution of multi-order interactions varies across the types of misclassification. For example, misclassified adversarial images have a higher strength of high-order interactions than correctly classified clean images, which indicates that adversarial perturbations create spurious features that arise from complex cooperation between pixels. By contrast, misclassified corrupted images have a lower strength of low-order interactions than correctly classified clean images, which indicates that corruptions break the local cooperation between pixels. We also provide the first analysis of Vision Transformers using interactions. We found that Vision Transformers show a different tendency in the distribution of interactions from that in CNNs, and this implies that they exploit the features that CNNs do not use for the prediction. Our study demonstrates that the recent game-theoretic analysis of deep learning models can be broadened to analyze various malfunctions of deep learning models including Vision Transformers by using the distribution, order, and sign of interactions.