Walker: Self-supervised Multiple Object Tracking by Walking on Temporal Appearance Graphs

TL;DR

Walker is a novel self-supervised multiple object tracking method that learns from videos with minimal annotations by leveraging temporal appearance graphs and contrastive learning, achieving competitive results without extensive labeling.

Contribution

It introduces the first self-supervised MOT framework using sparse annotations, combining graph-based learning and contrastive objectives for effective tracking.

Findings

Achieves competitive performance on MOT17, DanceTrack, and BDD100K.

Outperforms previous self-supervised trackers with up to 400x less annotation.

Effectively learns instance similarities without tracking labels.

Abstract

The supervision of state-of-the-art multiple object tracking (MOT) methods requires enormous annotation efforts to provide bounding boxes for all frames of all videos, and instance IDs to associate them through time. To this end, we introduce Walker, the first self-supervised tracker that learns from videos with sparse bounding box annotations, and no tracking labels. First, we design a quasi-dense temporal object appearance graph, and propose a novel multi-positive contrastive objective to optimize random walks on the graph and learn instance similarities. Then, we introduce an algorithm to enforce mutually-exclusive connective properties across instances in the graph, optimizing the learned topology for MOT. At inference time, we propose to associate detected instances to tracklets based on the max-likelihood transition state under motion-constrained bi-directional walks. Walker is the first self-supervised tracker to achieve competitive performance on MOT17, DanceTrack, and BDD100K. Remarkably, our proposal outperforms the previous self-supervised trackers even when drastically reducing the annotation requirements by up to 400x.