Detection-aware multi-object tracking evaluation

TL;DR

This paper introduces the Tracking Effort Measure (TEM), a new evaluation metric for multi-object tracking that fairly compares trackers using different detectors by quantifying the effort made by the tracker relative to its input data.

Contribution

The paper proposes TEM, a novel performance measure that evaluates tracker effort at frame and sequence levels, addressing fairness issues in tracker comparison.

Findings

TEM reduces correlation with input detection quality

TEM effectively quantifies tracker effort across datasets

Evaluation results highlight TEM's advantages over traditional measures

Abstract

How would you fairly evaluate two multi-object tracking algorithms (i.e. trackers), each one employing a different object detector? Detectors keep improving, thus trackers can make less effort to estimate object states over time. Is it then fair to compare a new tracker employing a new detector with another tracker using an old detector? In this paper, we propose a novel performance measure, named Tracking Effort Measure (TEM), to evaluate trackers that use different detectors. TEM estimates the improvement that the tracker does with respect to its input data (i.e. detections) at frame level (intra-frame complexity) and sequence level (inter-frame complexity). We evaluate TEM over well-known datasets, four trackers and eight detection sets. Results show that, unlike conventional tracking evaluation measures, TEM can quantify the effort done by the tracker with a reduced correlation on the input detections. Its implementation is publicly available online at https://github.com/vpulab/MOT-evaluation.