CrossTracker: Robust Multi-modal 3D Multi-Object Tracking via Cross Correction

TL;DR

CrossTracker introduces a two-stage multi-modal 3D multi-object tracking framework that refines trajectories by leveraging complementary camera and LiDAR data, significantly improving tracking robustness.

Contribution

It proposes a novel two-stage architecture with dedicated modules for multi-modal fusion, coarse trajectory generation, and cross correction-based refinement, enhancing multi-modal 3D MOT performance.

Findings

Outperforms 18 state-of-the-art competitors in experiments.

Effectively fuses camera and LiDAR data for robust tracking.

Demonstrates significant improvements in tracking accuracy and robustness.

Abstract

The fusion of camera- and LiDAR-based detections offers a promising solution to mitigate tracking failures in 3D multi-object tracking (MOT). However, existing methods predominantly exploit camera detections to correct tracking failures caused by potential LiDAR detection problems, neglecting the reciprocal benefit of refining camera detections using LiDAR data. This limitation is rooted in their single-stage architecture, akin to single-stage object detectors, lacking a dedicated trajectory refinement module to fully exploit the complementary multi-modal information. To this end, we introduce CrossTracker, a novel two-stage paradigm for online multi-modal 3D MOT. CrossTracker operates in a coarse-to-fine manner, initially generating coarse trajectories and subsequently refining them through an independent refinement process. Specifically, CrossTracker incorporates three essential modules: i) a multi-modal modeling (M^3) module that, by fusing multi-modal information (images, point clouds, and even plane geometry extracted from images), provides a robust metric for subsequent trajectory generation. ii) a coarse trajectory generation (C-TG) module that generates initial coarse dual-stream trajectories, and iii) a trajectory refinement (TR) module that refines coarse trajectories through cross correction between camera and LiDAR streams. Comprehensive experiments demonstrate the superior performance of our CrossTracker over its eighteen competitors, underscoring its effectiveness in harnessing the synergistic benefits of camera and LiDAR sensors for robust multi-modal 3D MOT.