Improving Visual Object Tracking through Visual Prompting

TL;DR

This paper introduces PiVOT, a novel visual prompting mechanism leveraging the CLIP foundation model to dynamically generate and refine prompts, significantly enhancing the discriminative ability of generic object trackers against distractors.

Contribution

The paper proposes a new visual prompting approach for object tracking that uses CLIP to automatically generate and refine prompts online, improving distractor suppression and tracking accuracy.

Findings

PiVOT improves tracking performance across multiple benchmarks.

The method effectively suppresses distractors during tracking.

Extensive experiments validate the effectiveness of the proposed prompting mechanism.

Abstract

Learning a discriminative model that distinguishes the specified target from surrounding distractors across frames is essential for generic object tracking (GOT). Dynamic adaptation of target representation against distractors remains challenging because prevailing trackers exhibit limited discriminative capability. To address this issue, we present a new visual prompting mechanism for generic object tracking, termed PiVOT. PiVOT introduces mechanisms that leverage the pretrained foundation model (CLIP) to automatically generate and refine visual prompts online, thereby enabling the tracker to suppress distractors through contrastive guidance. To transfer contrastive knowledge from the foundation model to the tracker, PiVOT automatically propagates this knowledge online and dynamically generates and updates visual prompts. Specifically, it proposes a prompt initialization mechanism that produces an initial visual prompt highlighting potential target locations. The foundation model is then used to refine the prompt based on appearance similarities between candidate objects and reference templates across potential targets. After refinement, the visual prompt better highlights potential target locations and reduces irrelevant prompt information. With the proposed prompting mechanism, the tracker can generate instance-aware feature maps guided by the visual prompts, which are incrementally and automatically updated during tracking, thereby effectively suppressing distractors. Extensive experiments across multiple benchmarks indicate that PiVOT, with the proposed prompting mechanism, can suppress distracting objects and improve tracking performance.