Learning to Track Instance from Single Nature Language Description

TL;DR

This paper introduces { racker}, a self-supervised vision-language tracker that learns to track objects from natural language descriptions without bounding-box annotations, using a novel dynamic token aggregation approach.

Contribution

It proposes a new self-supervised VL tracking method with a dynamic token aggregation module that improves semantic alignment and tracking performance without labeled data.

Findings

{ racker} surpasses state-of-the-art self-supervised methods on VL tracking benchmarks.

The dynamic token aggregation enhances semantic alignment between language and visual tokens.

The method enables effective instance tracking from unlabeled videos without bounding box annotations.

Abstract

How to achieve vision-language (VL) tracking using natural language descriptions from a video sequence \textbf{without relying on any bounding-box ground truth}? In this work, we achieve this goal by tackling \textit{self-supervised VL tracking}, which aims to evaluate tracking capabilities guided by natural language descriptions. We introduce \textbf{\tracker}, a novel self-supervised VL tracker that is capable of tracking any referred object by a language description. Unlike traditional methods that equally fuse all language and visual tokens, we propose an efficient Dynamic Token Aggregation Module, which treats each visual token \textbf{unequally}. The module consists of three main steps: i) Based on an anchor token, it selects multiple important target tokens from the template frame. ii) The selected target tokens are merged according to their attention scores and aggregated into the language tokens, thereby eliminating redundant visual token noise and enhancing semantic alignment. iii) Finally, the fused language tokens serve as guiding signals to extract potential target tokens from the search frame and propagate them to subsequent frames, enhancing temporal prompts and encouraging the tracker to autonomously learn instance tracking from unlabeled videos. This new modeling approach enables the effective self-supervised learning of language-guided tracking representations without the need for large-scale bounding box annotations. Extensive experiments on VL tracking benchmarks show that {\tracker} surpasses SOTA self-supervised methods.