MOTChallenge 2015: Towards a Benchmark for Multi-Target Tracking

TL;DR

This paper introduces MOTChallenge, a new benchmark for multi-target tracking that standardizes data collection, evaluation, and comparison of methods to advance research in the field.

Contribution

It presents a comprehensive framework for a unified multi-target tracking benchmark, addressing inconsistencies and establishing standardized evaluation procedures.

Findings

Developed a new dataset for multi-target tracking

Collected state-of-the-art tracking methods for testing

Created a unified evaluation system for benchmarking

Abstract

In the recent past, the computer vision community has developed centralized benchmarks for the performance evaluation of a variety of tasks, including generic object and pedestrian detection, 3D reconstruction, optical flow, single-object short-term tracking, and stereo estimation. Despite potential pitfalls of such benchmarks, they have proved to be extremely helpful to advance the state of the art in the respective area. Interestingly, there has been rather limited work on the standardization of quantitative benchmarks for multiple target tracking. One of the few exceptions is the well-known PETS dataset, targeted primarily at surveillance applications. Despite being widely used, it is often applied inconsistently, for example involving using different subsets of the available data, different ways of training the models, or differing evaluation scripts. This paper describes our work toward a novel multiple object tracking benchmark aimed to address such issues. We discuss the challenges of creating such a framework, collecting existing and new data, gathering state-of-the-art methods to be tested on the datasets, and finally creating a unified evaluation system. With MOTChallenge we aim to pave the way toward a unified evaluation framework for a more meaningful quantification of multi-target tracking.