Temporal-Enhanced Multimodal Transformer for Referring Multi-Object Tracking and Segmentation

TL;DR

This paper introduces TenRMOT, a Transformer-based method for referring multi-object tracking and segmentation that leverages multi-stage feature fusion, language-guided queries, and temporal priors, along with a new dataset, Ref-KITTI Segmentation.

Contribution

The paper proposes a novel Transformer-based framework with multi-stage feature fusion and temporal priors for RMOT, and introduces a new challenging dataset for referring multi-object tracking and segmentation.

Findings

TenRMOT outperforms existing methods on RMOT and segmentation tasks.

The new dataset Ref-KITTI Segmentation contains 18 videos with 818 expressions.

Temporal priors improve trajectory consistency in tracking.

Abstract

Referring multi-object tracking (RMOT) is an emerging cross-modal task that aims to locate an arbitrary number of target objects and maintain their identities referred by a language expression in a video. This intricate task involves the reasoning of linguistic and visual modalities, along with the temporal association of target objects. However, the seminal work employs only loose feature fusion and overlooks the utilization of long-term information on tracked objects. In this study, we introduce a compact Transformer-based method, termed TenRMOT. We conduct feature fusion at both encoding and decoding stages to fully exploit the advantages of Transformer architecture. Specifically, we incrementally perform cross-modal fusion layer-by-layer during the encoding phase. In the decoding phase, we utilize language-guided queries to probe memory features for accurate prediction of the desired objects. Moreover, we introduce a query update module that explicitly leverages temporal prior information of the tracked objects to enhance the consistency of their trajectories. In addition, we introduce a novel task called Referring Multi-Object Tracking and Segmentation (RMOTS) and construct a new dataset named Ref-KITTI Segmentation. Our dataset consists of 18 videos with 818 expressions, and each expression averages 10.7 masks, which poses a greater challenge compared to the typical single mask in most existing referring video segmentation datasets. TenRMOT demonstrates superior performance on both the referring multi-object tracking and the segmentation tasks.