The Origins and Prevalence of Texture Bias in Convolutional Neural Networks

TL;DR

This paper investigates the texture bias in CNNs, showing it is influenced by training data and augmentation, and that models can be trained to classify images more by shape than texture, aligning more with human perception.

Contribution

The study reveals how data augmentation and training objectives affect texture bias in CNNs and demonstrates methods to promote shape-based classification.

Findings

Data augmentation significantly reduces texture bias.

Models trained with naturalistic augmentations classify by shape more often.

Texture bias persists across different architectures and training objectives.

Abstract

Recent work has indicated that, unlike humans, ImageNet-trained CNNs tend to classify images by texture rather than by shape. How pervasive is this bias, and where does it come from? We find that, when trained on datasets of images with conflicting shape and texture, CNNs learn to classify by shape at least as easily as by texture. What factors, then, produce the texture bias in CNNs trained on ImageNet? Different unsupervised training objectives and different architectures have small but significant and largely independent effects on the level of texture bias. However, all objectives and architectures still lead to models that make texture-based classification decisions a majority of the time, even if shape information is decodable from their hidden representations. The effect of data augmentation is much larger. By taking less aggressive random crops at training time and applying simple, naturalistic augmentation (color distortion, noise, and blur), we train models that classify ambiguous images by shape a majority of the time, and outperform baselines on out-of-distribution test sets. Our results indicate that apparent differences in the way humans and ImageNet-trained CNNs process images may arise not primarily from differences in their internal workings, but from differences in the data that they see.