Humans disagree with the IoU for measuring object detector localization error

TL;DR

This paper reveals that human perception of object localization errors differs from IoU scores, suggesting IoU alone may not fully capture human judgment of localization quality.

Contribution

It is the first study to compare human judgments with IoU scores for object detector localization errors, highlighting the limitations of IoU as an evaluation metric.

Findings

Humans do not consider equal IoU scores as equally acceptable.

Participants show preferences for certain localization errors over others with the same IoU.

IoU scores alone may not align with human perception of localization quality.

Abstract

The localization quality of automatic object detectors is typically evaluated by the Intersection over Union (IoU) score. In this work, we show that humans have a different view on localization quality. To evaluate this, we conduct a survey with more than 70 participants. Results show that for localization errors with the exact same IoU score, humans might not consider that these errors are equal, and express a preference. Our work is the first to evaluate IoU with humans and makes it clear that relying on IoU scores alone to evaluate localization errors might not be sufficient.