IP-MOT: Instance Prompt Learning for Cross-Domain Multi-Object Tracking

TL;DR

IP-MOT introduces an innovative transformer-based approach that uses instance-level pseudo textual descriptions and a query-balanced strategy to enhance cross-domain multi-object tracking performance.

Contribution

The paper presents a novel end-to-end transformer model for MOT that leverages pseudo textual descriptions and a query-balanced strategy to improve cross-domain generalization.

Findings

Achieves competitive same-domain MOT performance.

Significantly improves cross-domain tracking accuracy.

Demonstrates effectiveness on multiple benchmark datasets.

Abstract

Multi-Object Tracking (MOT) aims to associate multiple objects across video frames and is a challenging vision task due to inherent complexities in the tracking environment. Most existing approaches train and track within a single domain, resulting in a lack of cross-domain generalizability to data from other domains. While several works have introduced natural language representation to bridge the domain gap in visual tracking, these textual descriptions often provide too high-level a view and fail to distinguish various instances within the same class. In this paper, we address this limitation by developing IP-MOT, an end-to-end transformer model for MOT that operates without concrete textual descriptions. Our approach is underpinned by two key innovations: Firstly, leveraging a pre-trained vision-language model, we obtain instance-level pseudo textual descriptions via prompt-tuning, which are invariant across different tracking scenes; Secondly, we introduce a query-balanced strategy, augmented by knowledge distillation, to further boost the generalization capabilities of our model. Extensive experiments conducted on three widely used MOT benchmarks, including MOT17, MOT20, and DanceTrack, demonstrate that our approach not only achieves competitive performance on same-domain data compared to state-of-the-art models but also significantly improves the performance of query-based trackers by large margins for cross-domain inputs.