Learning to Track Objects from Unlabeled Videos

TL;DR

This paper introduces an unsupervised learning method for single object tracking from unlabeled videos, addressing key challenges and achieving performance comparable to supervised methods.

Contribution

It proposes a three-stage unsupervised training approach including object discovery, naive training, and cycle memory learning for online update.

Findings

Outperforms existing unsupervised trackers significantly.

Achieves comparable results to supervised deep trackers.

Demonstrates effectiveness of cycle memory learning scheme.

Abstract

In this paper, we propose to learn an Unsupervised Single Object Tracker (USOT) from scratch. We identify that three major challenges, i.e., moving object discovery, rich temporal variation exploitation, and online update, are the central causes of the performance bottleneck of existing unsupervised trackers. To narrow the gap between unsupervised trackers and supervised counterparts, we propose an effective unsupervised learning approach composed of three stages. First, we sample sequentially moving objects with unsupervised optical flow and dynamic programming, instead of random cropping. Second, we train a naive Siamese tracker from scratch using single-frame pairs. Third, we continue training the tracker with a novel cycle memory learning scheme, which is conducted in longer temporal spans and also enables our tracker to update online. Extensive experiments show that the proposed USOT learned from unlabeled videos performs well over the state-of-the-art unsupervised trackers by large margins, and on par with recent supervised deep trackers. Code is available at https://github.com/VISION-SJTU/USOT.