GSOT3D: Towards Generic 3D Single Object Tracking in the Wild

TL;DR

GSOT3D introduces the largest benchmark dataset for generic 3D single object tracking in diverse real-world scenarios, enabling comprehensive evaluation and fostering development of more robust tracking algorithms.

Contribution

The paper presents GSOT3D, a new large-scale benchmark with multi-modal data for 3D object tracking, and proposes PROT3D, a novel tracking method that outperforms existing models.

Findings

Existing models perform poorly on GSOT3D, indicating room for improvement.

PROT3D significantly outperforms current solutions in 3D tracking accuracy.

GSOT3D facilitates diverse research directions in 3D object tracking.

Abstract

In this paper, we present a novel benchmark, GSOT3D, that aims at facilitating development of generic 3D single object tracking (SOT) in the wild. Specifically, GSOT3D offers 620 sequences with 123K frames, and covers a wide selection of 54 object categories. Each sequence is offered with multiple modalities, including the point cloud (PC), RGB image, and depth. This allows GSOT3D to support various 3D tracking tasks, such as single-modal 3D SOT on PC and multi-modal 3D SOT on RGB-PC or RGB-D, and thus greatly broadens research directions for 3D object tracking. To provide highquality per-frame 3D annotations, all sequences are labeled manually with multiple rounds of meticulous inspection and refinement. To our best knowledge, GSOT3D is the largest benchmark dedicated to various generic 3D object tracking tasks. To understand how existing 3D trackers perform and to provide comparisons for future research on GSOT3D, we assess eight representative point cloud-based tracking models. Our evaluation results exhibit that these models heavily degrade on GSOT3D, and more efforts are required for robust and generic 3D object tracking. Besides, to encourage future research, we present a simple yet effective generic 3D tracker, named PROT3D, that localizes the target object via a progressive spatial-temporal network and outperforms all current solutions by a large margin. By releasing GSOT3D, we expect to advance further 3D tracking in future research and applications. Our benchmark and model as well as the evaluation results will be publicly released at our webpage https://github.com/ailovejinx/GSOT3D.