On the Effects of Artificial Data Modification

TL;DR

This paper investigates how artificial data modifications like augmentation artifacts influence vision model training and evaluation, revealing biases in current assessment methods and advocating for understanding and leveraging these effects rather than eliminating them.

Contribution

It uncovers biases in shape bias and robustness measures, proposes a fairer evaluation method, and emphasizes the importance of understanding artificial data artifacts in vision models.

Findings

Current shape bias identification methods are biased.

Proposed a fairer alternative for occlusion robustness measurement.

Artificial data artifacts can be beneficial and should be exploited.

Abstract

Data distortion is commonly applied in vision models during both training (e.g methods like MixUp and CutMix) and evaluation (e.g. shape-texture bias and robustness). This data modification can introduce artificial information. It is often assumed that the resulting artefacts are detrimental to training, whilst being negligible when analysing models. We investigate these assumptions and conclude that in some cases they are unfounded and lead to incorrect results. Specifically, we show current shape bias identification methods and occlusion robustness measures are biased and propose a fairer alternative for the latter. Subsequently, through a series of experiments we seek to correct and strengthen the community's perception of how augmenting affects learning of vision models. Based on our empirical results we argue that the impact of the artefacts must be understood and exploited rather than eliminated.