Multi-Object Tracking with Hallucinated and Unlabeled Videos

TL;DR

This paper introduces a novel weakly supervised deep neural tracker trained without tracking annotations by hallucinating videos from images and mining hard examples from unlabeled videos, achieving state-of-the-art results.

Contribution

It presents a new method for training deep trackers using hallucinated videos and unlabeled data, reducing reliance on costly annotations.

Findings

Achieved state-of-the-art performance on MOT17 and TAO-person datasets.

Combined hallucinated data with real annotations for improved results.

Demonstrated effectiveness of hard example mining from unlabeled videos.

Abstract

In this paper, we explore learning end-to-end deep neural trackers without tracking annotations. This is important as large-scale training data is essential for training deep neural trackers while tracking annotations are expensive to acquire. In place of tracking annotations, we first hallucinate videos from images with bounding box annotations using zoom-in/out motion transformations to obtain free tracking labels. We add video simulation augmentations to create a diverse tracking dataset, albeit with simple motion. Next, to tackle harder tracking cases, we mine hard examples across an unlabeled pool of real videos with a tracker trained on our hallucinated video data. For hard example mining, we propose an optimization-based connecting process to first identify and then rectify hard examples from the pool of unlabeled videos. Finally, we train our tracker jointly on hallucinated data and mined hard video examples. Our weakly supervised tracker achieves state-of-the-art performance on the MOT17 and TAO-person datasets. On MOT17, we further demonstrate that the combination of our self-generated data and the existing manually-annotated data leads to additional improvements.