RegTrack: Simplicity Beneath Complexity in Robust Multi-Modal 3D Multi-Object Tracking

TL;DR

RegTrack introduces a simple yet effective multi-modal 3D multi-object tracking method that leverages a unified tri-cue encoder inspired by gauge theory, achieving superior robustness and efficiency without complex association metrics.

Contribution

It proposes RegTrack, a novel 3D MOT approach that uses a tri-cue encoder and pairwise similarity-based association, challenging the need for complex, class-specific priors.

Findings

Outperforms 35 competitors on KITTI and nuScenes datasets.

Uses only 2.6 million parameters, demonstrating efficiency.

Achieves robust and generalizable tracking with point cloud inputs.

Abstract

Existing 3D multi-object tracking (MOT) methods often sacrifice efficiency and generalizability for robustness, largely relying on complex association metrics derived from multi-modal architectures and class-specific motion priors. Challenging the rooted belief that greater complexity necessarily yields greater robustness, we propose a robust, efficient, and generalizable method for multi-modal 3D MOT, dubbed RegTrack. Inspired by Yang-Mills gauge theory, RegTrack is built upon a unified tri-cue encoder (UTEnc), comprising three tightly coupled components: a local-global point cloud encoder (LG-PEnc), a mixture-of-experts-based geometry encoder (MoE-GEnc), and an image encoder from a well-pretrained visual-language model. LG-PEnc efficiently encodes the spatial and structural information of point clouds to produce foundational representations for each object, whose pairwise similarities serve as the sole association metric. MoE-GEnc seamlessly interacts with LG-PEnc to model inter-object geometric relationships across frames, adaptively compensating for inter-frame object motion without relying on any class-specific priors. The image encoder is kept frozen and is used exclusively during training to provide a well-pretrained representation space. Point cloud representations are aligned to this space to supervise the motion compensation process, encouraging representation invariance across frames for the same object while enhancing discriminability among different objects. Through this formulation, RegTrack attains robust, efficient, and generalizable inference using only point cloud inputs, requiring just 2.6M parameters. Extensive experiments on KITTI and nuScenes show that RegTrack outperforms its thirty-five competitors.