Prompting for Multi-Modal Tracking

TL;DR

This paper introduces ProTrack, a novel multi-modal tracking approach that leverages visual prompts to convert multi-modal data into a single modality, enabling high-performance tracking without additional multi-modal training.

Contribution

ProTrack is a new multi-modal tracking method that uses prompts to transfer multi-modal inputs into a single modality, avoiding the need for multi-modal fusion modules or extra training.

Findings

ProTrack achieves state-of-the-art results on 5 benchmark datasets.

It effectively utilizes pre-trained RGB trackers for multi-modal tracking.

The approach requires no additional training on multi-modal data.

Abstract

Multi-modal tracking gains attention due to its ability to be more accurate and robust in complex scenarios compared to traditional RGB-based tracking. Its key lies in how to fuse multi-modal data and reduce the gap between modalities. However, multi-modal tracking still severely suffers from data deficiency, thus resulting in the insufficient learning of fusion modules. Instead of building such a fusion module, in this paper, we provide a new perspective on multi-modal tracking by attaching importance to the multi-modal visual prompts. We design a novel multi-modal prompt tracker (ProTrack), which can transfer the multi-modal inputs to a single modality by the prompt paradigm. By best employing the tracking ability of pre-trained RGB trackers learning at scale, our ProTrack can achieve high-performance multi-modal tracking by only altering the inputs, even without any extra training on multi-modal data. Extensive experiments on 5 benchmark datasets demonstrate the effectiveness of the proposed ProTrack.