Benchmark for Anonymous Video Analytics

TL;DR

This paper introduces the first comprehensive benchmark dataset and evaluation framework for assessing computer vision-based out-of-home audience measurement methods, enabling standardized comparison of algorithms and commercial solutions.

Contribution

It provides a novel dataset, performance metrics, and a comparative analysis of multiple algorithms and commercial solutions for audience localization, counting, and demographics.

Findings

Open-source algorithms vary significantly in performance.

Commercial solutions outperform some open-source methods in certain tasks.

Benchmark facilitates standardized evaluation of audience measurement techniques.

Abstract

Out-of-home audience measurement aims to count and characterize the people exposed to advertising content in the physical world. While audience measurement solutions based on computer vision are of increasing interest, no commonly accepted benchmark exists to evaluate and compare their performance. In this paper, we propose the first benchmark for digital out-of-home audience measurement that evaluates the vision-based tasks of audience localization and counting, and audience demographics. The benchmark is composed of a novel, dataset captured at multiple locations and a set of performance measures. Using the benchmark, we present an in-depth comparison of eight open-source algorithms on four hardware platforms with GPU and CPU-optimized inferences and of two commercial off-the-shelf solutions for localization, count, age, and gender estimation. This benchmark and related open-source codes are available at http://ava.eecs.qmul.ac.uk.