I Am Big, You Are Little; I Am Right, You Are Wrong

TL;DR

This paper investigates how different image classification models focus on pixels by analyzing minimal pixel sets, revealing architecture-dependent concentration patterns and links to misclassification.

Contribution

It introduces the concept of minimal sufficient pixel sets to measure model concentration, providing new insights into model decision processes across architectures.

Findings

Different architectures have statistically distinct pixel concentration patterns.

ConvNext and EVA models differ significantly from others in pixel focus.

Misclassified images are associated with larger pixel sets.

Abstract

Machine learning for image classification is an active and rapidly developing field. With the proliferation of classifiers of different sizes and different architectures, the problem of choosing the right model becomes more and more important. While we can assess a model's classification accuracy statistically, our understanding of the way these models work is unfortunately limited. In order to gain insight into the decision-making process of different vision models, we propose using minimal sufficient pixels sets to gauge a model's `concentration': the pixels that capture the essence of an image through the lens of the model. By comparing position, overlap, and size of sets of pixels, we identify that different architectures have statistically different concentration, in both size and position. In particular, ConvNext and EVA models differ markedly from the others. We also identify that images which are misclassified are associated with larger pixels sets than correct classifications.