Bootstrapping Referring Multi-Object Tracking

TL;DR

This paper introduces a new task called referring multi-object tracking (RMOT) that uses language cues to track multiple objects over time, along with a benchmark and a Transformer-based model that achieves state-of-the-art results.

Contribution

The paper proposes RMOT as a novel task, creates the Refer-KITTI-V2 benchmark, and develops TempRMOT, a Transformer-based framework for effective multi-object tracking guided by language.

Findings

TempRMOT outperforms previous methods on Refer-KITTI and Refer-KITTI-V2

The semi-automatic labeling pipeline efficiently generates diverse annotations

TempRMOT effectively models long-term spatial-temporal object interactions

Abstract

Referring understanding is a fundamental task that bridges natural language and visual content by localizing objects described in free-form expressions. However, existing works are constrained by limited language expressiveness, lacking the capacity to model object dynamics in spatial numbers and temporal states. To address these limitations, we introduce a new and general referring understanding task, termed referring multi-object tracking (RMOT). Its core idea is to employ a language expression as a semantic cue to guide the prediction of multi-object tracking, comprehensively accounting for variations in object quantity and temporal semantics. Along with RMOT, we introduce a RMOT benchmark named Refer-KITTI-V2, featuring scalable and diverse language expressions. To efficiently generate high-quality annotations covering object dynamics with minimal manual effort, we propose a semi-automatic labeling pipeline that formulates a total of 9,758 language prompts. In addition, we propose TempRMOT, an elegant end-to-end Transformer-based framework for RMOT. At its core is a query-driven Temporal Enhancement Module that represents each object as a Transformer query, enabling long-term spatial-temporal interactions with other objects and past frames to efficiently refine these queries. TempRMOT achieves state-of-the-art performance on both Refer-KITTI and Refer-KITTI-V2, demonstrating the effectiveness of our approach. The source code and dataset is available at https://github.com/zyn213/TempRMOT.