Err on the Side of Texture: Texture Bias on Real Data

TL;DR

This paper introduces the Texture Association Value (TAV), a new metric to quantify texture reliance in image classification models, revealing that texture bias significantly affects model accuracy and robustness, especially in adversarial examples.

Contribution

The paper presents TAV as a novel metric for measuring texture bias and demonstrates its impact on model robustness and accuracy in real-world data.

Findings

Over 90% of natural adversarial examples contain misleading textures.

Texture bias correlates strongly with model misclassification.

Texture reliance affects both accuracy and robustness of models.

Abstract

Bias significantly undermines both the accuracy and trustworthiness of machine learning models. To date, one of the strongest biases observed in image classification models is texture bias-where models overly rely on texture information rather than shape information. Yet, existing approaches for measuring and mitigating texture bias have not been able to capture how textures impact model robustness in real-world settings. In this work, we introduce the Texture Association Value (TAV), a novel metric that quantifies how strongly models rely on the presence of specific textures when classifying objects. Leveraging TAV, we demonstrate that model accuracy and robustness are heavily influenced by texture. Our results show that texture bias explains the existence of natural adversarial examples, where over 90% of these samples contain textures that are misaligned with the learned texture of their true label, resulting in confident mispredictions.