MLS-Track: Multilevel Semantic Interaction in RMOT

TL;DR

This paper introduces MLS-Track, a novel multi-level semantic interaction framework for language-based multi-object tracking, supported by a new synthetic dataset, Refer-UE-City, to overcome data scarcity and achieve state-of-the-art results.

Contribution

The paper presents MLS-Track, a multi-level semantic-guided framework, and constructs Refer-UE-City, a synthetic dataset for language-based multi-object tracking.

Findings

MLS-Track achieves state-of-the-art performance on Refer-UE-City and Refer-KITTI.

The proposed dataset and framework effectively address data scarcity in language-based tracking.

Semantic Guidance Module and Semantic Correlation Branch enhance interaction between text and model layers.

Abstract

The new trend in multi-object tracking task is to track objects of interest using natural language. However, the scarcity of paired prompt-instance data hinders its progress. To address this challenge, we propose a high-quality yet low-cost data generation method base on Unreal Engine 5 and construct a brand-new benchmark dataset, named Refer-UE-City, which primarily includes scenes from intersection surveillance videos, detailing the appearance and actions of people and vehicles. Specifically, it provides 14 videos with a total of 714 expressions, and is comparable in scale to the Refer-KITTI dataset. Additionally, we propose a multi-level semantic-guided multi-object framework called MLS-Track, where the interaction between the model and text is enhanced layer by layer through the introduction of Semantic Guidance Module (SGM) and Semantic Correlation Branch (SCB). Extensive experiments on Refer-UE-City and Refer-KITTI datasets demonstrate the effectiveness of our proposed framework and it achieves state-of-the-art performance. Code and datatsets will be available.