Sensitivity of Average Precision to Bounding Box Perturbations

TL;DR

This paper investigates how small perturbations to bounding boxes, such as pixel shifts, significantly impact the Average Precision score in object detection, revealing sensitivity issues that affect model evaluation.

Contribution

It provides a quantitative analysis of AP sensitivity to bounding box perturbations, highlighting the challenge in improving mAP scores as models advance.

Findings

AP drops by 8.4% with a one-pixel shift

Small objects experience a 23.1% mAP drop with one-pixel shift

Ground-truth boxes are also highly sensitive to perturbations

Abstract

Object detection is a fundamental vision task. It has been highly researched in academia and has been widely adopted in industry. Average Precision (AP) is the standard score for evaluating object detectors. Our understanding of the subtleties of this score, however, is limited. Here, we quantify the sensitivity of AP to bounding box perturbations and show that AP is very sensitive to small translations. Only one pixel shift is enough to drop the mAP of a model by 8.4%. The mAP drop over small objects with only one pixel shift is 23.1%. The corresponding numbers when ground-truth (GT) boxes are used as predictions are 23% and 41.7%, respectively. These results explain why achieving higher mAP becomes increasingly harder as models get better. We also investigate the effect of box scaling on AP. Code and data is available at https://github.com/aliborji/AP_Box_Perturbation.