End-to-end Evaluation of Practical Video Analytics Systems for Face Detection and Recognition

TL;DR

This paper evaluates a practical, end-to-end face detection and recognition system in bandwidth-constrained environments, highlighting issues with independent task evaluation and dataset imbalance, and proposing solutions for more accurate performance assessment.

Contribution

It introduces a comprehensive end-to-end evaluation framework for face analytics systems, addressing dataset imbalance and annotation inconsistencies to improve performance estimates.

Findings

End-to-end evaluation yields more accurate performance metrics.

Dataset balancing and annotation consistency are crucial for reliable assessment.

Sequential task evaluation reveals interdependencies affecting system performance.

Abstract

Practical video analytics systems that are deployed in bandwidth constrained environments like autonomous vehicles perform computer vision tasks such as face detection and recognition. In an end-to-end face analytics system, inputs are first compressed using popular video codecs like HEVC and then passed onto modules that perform face detection, alignment, and recognition sequentially. Typically, the modules of these systems are evaluated independently using task-specific imbalanced datasets that can misconstrue performance estimates. In this paper, we perform a thorough end-to-end evaluation of a face analytics system using a driving-specific dataset, which enables meaningful interpretations. We demonstrate how independent task evaluations, dataset imbalances, and inconsistent annotations can lead to incorrect system performance estimates. We propose strategies to create balanced evaluation subsets of our dataset and to make its annotations consistent across multiple analytics tasks and scenarios. We then evaluate the end-to-end system performance sequentially to account for task interdependencies. Our experiments show that our approach provides consistent, accurate, and interpretable estimates of the system's performance which is critical for real-world applications.