Track to Reconstruct and Reconstruct to Track

TL;DR

This paper introduces MOTSFusion, a novel method that integrates object tracking and 3D reconstruction in a loop, improving long-term tracking accuracy by leveraging dynamic 3D object motion.

Contribution

The paper presents a new approach that combines tracking and reconstruction, enabling better handling of occlusions and missing detections through dynamic 3D object reconstructions.

Findings

Reduces ID switches by over 50% on KITTI dataset

Outperforms previous methods in bounding box and segmentation tracking

Effectively handles occlusions and missing detections

Abstract

Object tracking and 3D reconstruction are often performed together, with tracking used as input for reconstruction. However, the obtained reconstructions also provide useful information for improving tracking. We propose a novel method that closes this loop, first tracking to reconstruct, and then reconstructing to track. Our approach, MOTSFusion (Multi-Object Tracking, Segmentation and dynamic object Fusion), exploits the 3D motion extracted from dynamic object reconstructions to track objects through long periods of complete occlusion and to recover missing detections. Our approach first builds up short tracklets using 2D optical flow, and then fuses these into dynamic 3D object reconstructions. The precise 3D object motion of these reconstructions is used to merge tracklets through occlusion into long-term tracks, and to locate objects when detections are missing. On KITTI, our reconstruction-based tracking reduces the number of ID switches of the initial tracklets by more than 50%, and outperforms all previous approaches for both bounding box and segmentation tracking.