MUTR3D: A Multi-camera Tracking Framework via 3D-to-2D Queries

TL;DR

MUTR3D introduces a novel multi-camera 3D tracking framework that models object trajectories using 3D track queries, eliminating the need for explicit appearance or spatial similarity measures, and achieves superior performance on nuScenes.

Contribution

It proposes a new end-to-end 3D tracking method using 3D track queries linked with 2D camera observations, avoiding traditional association techniques.

Findings

Outperforms state-of-the-art by 5.3 AMOTA on nuScenes

Does not require post-processing like NMS or bounding box association

Uses set-to-set loss for direct comparison with ground truth

Abstract

Accurate and consistent 3D tracking from multiple cameras is a key component in a vision-based autonomous driving system. It involves modeling 3D dynamic objects in complex scenes across multiple cameras. This problem is inherently challenging due to depth estimation, visual occlusions, appearance ambiguity, etc. Moreover, objects are not consistently associated across time and cameras. To address that, we propose an end-to-end \textbf{MU}lti-camera \textbf{TR}acking framework called MUTR3D. In contrast to prior works, MUTR3D does not explicitly rely on the spatial and appearance similarity of objects. Instead, our method introduces \textit{3D track query} to model spatial and appearance coherent track for each object that appears in multiple cameras and multiple frames. We use camera transformations to link 3D trackers with their observations in 2D images. Each tracker is further refined according to the features that are obtained from camera images. MUTR3D uses a set-to-set loss to measure the difference between the predicted tracking results and the ground truths. Therefore, it does not require any post-processing such as non-maximum suppression and/or bounding box association. MUTR3D outperforms state-of-the-art methods by 5.3 AMOTA on the nuScenes dataset. Code is available at: \url{https://github.com/a1600012888/MUTR3D}.