Joint Spatial-Temporal Optimization for Stereo 3D Object Tracking

TL;DR

This paper introduces a joint spatial-temporal optimization method for stereo 3D object tracking that combines deep learning and geometric modeling to improve accuracy and consistency in tracking multiple objects in 3D space.

Contribution

It proposes a novel joint optimization framework that integrates deep neural network cues with geometric modeling for more accurate 3D object tracking.

Findings

Outperforms previous image-based 3D tracking methods on KITTI dataset.

Effectively models object motion and localization using joint spatial-temporal optimization.

Demonstrates robustness across multiple datasets and categories.

Abstract

Directly learning multiple 3D objects motion from sequential images is difficult, while the geometric bundle adjustment lacks the ability to localize the invisible object centroid. To benefit from both the powerful object understanding skill from deep neural network meanwhile tackle precise geometry modeling for consistent trajectory estimation, we propose a joint spatial-temporal optimization-based stereo 3D object tracking method. From the network, we detect corresponding 2D bounding boxes on adjacent images and regress an initial 3D bounding box. Dense object cues (local depth and local coordinates) that associating to the object centroid are then predicted using a region-based network. Considering both the instant localization accuracy and motion consistency, our optimization models the relations between the object centroid and observed cues into a joint spatial-temporal error function. All historic cues will be summarized to contribute to the current estimation by a per-frame marginalization strategy without repeated computation. Quantitative evaluation on the KITTI tracking dataset shows our approach outperforms previous image-based 3D tracking methods by significant margins. We also report extensive results on multiple categories and larger datasets (KITTI raw and Argoverse Tracking) for future benchmarking.