Per-frame mAP Prediction for Continuous Performance Monitoring of Object Detection During Deployment

TL;DR

This paper introduces a method for real-time prediction of object detection performance on a per-frame basis during deployment, enabling safer operation without ground truth data by monitoring internal detector features.

Contribution

It presents a novel introspection approach that predicts per-frame mAP drops in object detectors during deployment, addressing the limitations of static evaluation metrics.

Findings

Effective in predicting performance drops without ground truth

Reduces risk by timely raising alarms for low-performance frames

Demonstrates robustness across varying environmental conditions

Abstract

Performance monitoring of object detection is crucial for safety-critical applications such as autonomous vehicles that operate under varying and complex environmental conditions. Currently, object detectors are evaluated using summary metrics based on a single dataset that is assumed to be representative of all future deployment conditions. In practice, this assumption does not hold, and the performance fluctuates as a function of the deployment conditions. To address this issue, we propose an introspection approach to performance monitoring during deployment without the need for ground truth data. We do so by predicting when the per-frame mean average precision drops below a critical threshold using the detector's internal features. We quantitatively evaluate and demonstrate our method's ability to reduce risk by trading off making an incorrect decision by raising the alarm and absenting from detection.