TL;DR
BiasBed provides a comprehensive evaluation framework for texture and style bias in CNNs, highlighting the importance of rigorous testing and revealing that some existing methods lack significant improvements.
Contribution
Introduces BiasBed, a testbed with datasets and protocols for evaluating texture and style bias mitigation methods in neural networks.
Findings
Some algorithms do not significantly reduce style bias.
Rigorous statistical testing is essential for fair comparison.
Evaluation protocols impact perceived effectiveness of bias mitigation methods.
Abstract
The well-documented presence of texture bias in modern convolutional neural networks has led to a plethora of algorithms that promote an emphasis on shape cues, often to support generalization to new domains. Yet, common datasets, benchmarks and general model selection strategies are missing, and there is no agreed, rigorous evaluation protocol. In this paper, we investigate difficulties and limitations when training networks with reduced texture bias. In particular, we also show that proper evaluation and meaningful comparisons between methods are not trivial. We introduce BiasBed, a testbed for texture- and style-biased training, including multiple datasets and a range of existing algorithms. It comes with an extensive evaluation protocol that includes rigorous hypothesis testing to gauge the significance of the results, despite the considerable training instability of some style bias…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Machine Learning and Data Classification · Advanced Neural Network Applications
