An Informative Tracking Benchmark

TL;DR

This paper introduces a small, carefully selected tracking benchmark (ITB) that efficiently evaluates trackers across diverse challenging scenarios, reducing redundancy and improving assessment effectiveness.

Contribution

We propose a principled method to construct an informative, balanced, and diverse tracking benchmark using a quality assessment mechanism and additional collected sequences.

Findings

ITB covers all typical challenging scenarios with only 7% of original data

Analysis of 15 state-of-the-art trackers reveals effective methods for robustness

New challenges identified for future visual tracking research

Abstract

Along with the rapid progress of visual tracking, existing benchmarks become less informative due to redundancy of samples and weak discrimination between current trackers, making evaluations on all datasets extremely time-consuming. Thus, a small and informative benchmark, which covers all typical challenging scenarios to facilitate assessing the tracker performance, is of great interest. In this work, we develop a principled way to construct a small and informative tracking benchmark (ITB) with 7% out of 1.2 M frames of existing and newly collected datasets, which enables efficient evaluation while ensuring effectiveness. Specifically, we first design a quality assessment mechanism to select the most informative sequences from existing benchmarks taking into account 1) challenging level, 2) discriminative strength, 3) and density of appearance variations. Furthermore, we collect additional sequences to ensure the diversity and balance of tracking scenarios, leading to a total of 20 sequences for each scenario. By analyzing the results of 15 state-of-the-art trackers re-trained on the same data, we determine the effective methods for robust tracking under each scenario and demonstrate new challenges for future research direction in this field.