Decoupled Spatio-Temporal Consistency Learning for Self-Supervised Tracking

TL;DR

This paper introduces racker, a self-supervised tracking framework that learns from unlabeled videos by decoupling spatial and temporal consistency, significantly reducing the need for manual annotations while achieving state-of-the-art results.

Contribution

The paper proposes a novel decoupled spatio-temporal consistency training framework and an instance contrastive loss for self-supervised visual tracking, eliminating the need for box annotations.

Findings

Outperforms state-of-the-art self-supervised tracking methods.

Achieves over 25% improvement in AUC on GOT10K.

Demonstrates strong generalization across nine benchmark datasets.

Abstract

The success of visual tracking has been largely driven by datasets with manual box annotations. However, these box annotations require tremendous human effort, limiting the scale and diversity of existing tracking datasets. In this work, we present a novel Self-Supervised Tracking framework named \textbf{{\tracker}}, designed to eliminate the need of box annotations. Specifically, a decoupled spatio-temporal consistency training framework is proposed to learn rich target information across timestamps through global spatial localization and local temporal association. This allows for the simulation of appearance and motion variations of instances in real-world scenarios. Furthermore, an instance contrastive loss is designed to learn instance-level correspondences from a multi-view perspective, offering robust instance supervision without additional labels. This new design paradigm enables {\tracker} to effectively learn generic tracking representations in a self-supervised manner, while reducing reliance on extensive box annotations. Extensive experiments on nine benchmark datasets demonstrate that {\tracker} surpasses \textit{SOTA} self-supervised tracking methods, achieving an improvement of more than 25.3\%, 20.4\%, and 14.8\% in AUC (AO) score on the GOT10K, LaSOT, TrackingNet datasets, respectively. Code: https://github.com/GXNU-ZhongLab/SSTrack.