Visual object tracking performance measures revisited

TL;DR

This paper critically analyzes existing visual tracking performance measures, identifies their limitations, and proposes a simplified, more effective evaluation framework based on two core measures: accuracy and robustness.

Contribution

It revisits and theoretically evaluates current measures, demonstrating their equivalences and weaknesses, and proposes a standardized evaluation approach using two complementary measures.

Findings

Several measures are equivalent in information content

Some measures are more brittle and less reliable

Proposed measures are intuitive, visualizable, and adopted by VOT challenges

Abstract

The problem of visual tracking evaluation is sporting a large variety of performance measures, and largely suffers from lack of consensus about which measures should be used in experiments. This makes the cross-paper tracker comparison difficult. Furthermore, as some measures may be less effective than others, the tracking results may be skewed or biased towards particular tracking aspects. In this paper we revisit the popular performance measures and tracker performance visualizations and analyze them theoretically and experimentally. We show that several measures are equivalent from the point of information they provide for tracker comparison and, crucially, that some are more brittle than the others. Based on our analysis we narrow down the set of potential measures to only two complementary ones, describing accuracy and robustness, thus pushing towards homogenization of the tracker evaluation methodology. These two measures can be intuitively interpreted and visualized and have been employed by the recent Visual Object Tracking (VOT) challenges as the foundation for the evaluation methodology.