AUPIMO: Redefining Visual Anomaly Detection Benchmarks with High Speed and Low Tolerance
Joao P. C. Bertoldo, Dick Ameln, Ashwin Vaidya, Samet, Ak\c{c}ay

TL;DR
This paper introduces PIMO, a new evaluation metric for visual anomaly detection that addresses limitations of existing metrics, providing more robust, faster, and nuanced performance assessment, challenging the notion that current benchmarks are near-solved.
Contribution
We propose PIMO, a novel per-image recall-based metric that improves robustness, computational efficiency, and evaluation nuance in visual anomaly detection benchmarks.
Findings
PIMO accelerates computation compared to AUROC and AUPRO.
PIMO provides more robust performance evaluation under noisy annotations.
PIMO reveals that current models do not fully solve MVTec and VisA datasets.
Abstract
Recent advances in visual anomaly detection research have seen AUROC and AUPRO scores on public benchmark datasets such as MVTec and VisA converge towards perfect recall, giving the impression that these benchmarks are near-solved. However, high AUROC and AUPRO scores do not always reflect qualitative performance, which limits the validity of these metrics in real-world applications. We argue that the artificial ceiling imposed by the lack of an adequate evaluation metric restrains progression of the field, and it is crucial that we revisit the evaluation metrics used to rate our algorithms. In response, we introduce Per-IMage Overlap (PIMO), a novel metric that addresses the shortcomings of AUROC and AUPRO. PIMO retains the recall-based nature of the existing metrics but introduces two distinctions: the assignment of curves (and respective area under the curve) is per-image, and its…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAnomaly Detection Techniques and Applications · COVID-19 diagnosis using AI · Data-Driven Disease Surveillance
