Connecting metrics for shape-texture knowledge in computer vision

TL;DR

This paper investigates how different neural network architectures rely on shape versus texture features in image classification, revealing correlations and competition between these features and their impact on model performance.

Contribution

It extends analysis of shape and texture bias across many architectures, showing their correlation with performance and the anti-correlation between shape and texture representations.

Findings

Model performance correlates with shape bias at output and penultimate layer.

Shape and texture representations are strongly anti-correlated.

Architecture families show significant variation in shape bias and performance.

Abstract

Modern artificial neural networks, including convolutional neural networks and vision transformers, have mastered several computer vision tasks, including object recognition. However, there are many significant differences between the behavior and robustness of these systems and of the human visual system. Deep neural networks remain brittle and susceptible to many changes in the image that do not cause humans to misclassify images. Part of this different behavior may be explained by the type of features humans and deep neural networks use in vision tasks. Humans tend to classify objects according to their shape while deep neural networks seem to rely mostly on texture. Exploring this question is relevant, since it may lead to better performing neural network architectures and to a better understanding of the workings of the vision system of primates. In this work, we advance the state of the art in our understanding of this phenomenon, by extending previous analyses to a much larger set of deep neural network architectures. We found that the performance of models in image classification tasks is highly correlated with their shape bias measured at the output and penultimate layer. Furthermore, our results showed that the number of neurons that represent shape and texture are strongly anti-correlated, thus providing evidence that there is competition between these two types of features. Finally, we observed that while in general there is a correlation between performance and shape bias, there are significant variations between architecture families.