On Pitfalls of Measuring Occlusion Robustness through Data Distortion

TL;DR

This paper highlights the biases introduced by data distortion methods in measuring occlusion robustness and proposes iOcclusion as a fairer evaluation approach for real-world scenarios.

Contribution

It identifies pitfalls in current data distortion techniques for occlusion robustness and introduces iOcclusion to improve evaluation fairness.

Findings

Distorting images without considering artefacts biases robustness results.

Current evaluation methods may overestimate model performance under occlusion.

iOcclusion provides a more reliable assessment of occlusion robustness.

Abstract

Over the past years, the crucial role of data has largely been shadowed by the field's focus on architectures and training procedures. We often cause changes to the data without being aware of their wider implications. In this paper we show that distorting images without accounting for the artefacts introduced leads to biased results when establishing occlusion robustness. To ensure models behave as expected in real-world scenarios, we need to rule out the impact added artefacts have on evaluation. We propose a new approach, iOcclusion, as a fairer alternative for applications where the possible occluders are unknown.