3D Multi-Object Tracking with Differentiable Pose Estimation

TL;DR

This paper introduces a novel end-to-end method combining differentiable pose estimation and joint reconstruction for 3D multi-object tracking in indoor RGB-D sequences, significantly improving tracking accuracy.

Contribution

It presents a new graph-based, differentiable approach for joint 3D pose estimation and reconstruction, enhancing multi-object tracking robustness and consistency.

Findings

Improves MOTA score by 24.8% over state-of-the-art methods.

Introduces a synthetic dataset with 2381 sequences for evaluation.

Demonstrates significant performance boost in synthetic and real-world tests.

Abstract

We propose a novel approach for joint 3D multi-object tracking and reconstruction from RGB-D sequences in indoor environments. To this end, we detect and reconstruct objects in each frame while predicting dense correspondences mappings into a normalized object space. We leverage those correspondences to inform a graph neural network to solve for the optimal, temporally-consistent 7-DoF pose trajectories of all objects. The novelty of our method is two-fold: first, we propose a new graph-based approach for differentiable pose estimation over time to learn optimal pose trajectories; second, we present a joint formulation of reconstruction and pose estimation along the time axis for robust and geometrically consistent multi-object tracking. In order to validate our approach, we introduce a new synthetic dataset comprising 2381 unique indoor sequences with a total of 60k rendered RGB-D images for multi-object tracking with moving objects and camera positions derived from the synthetic 3D-FRONT dataset. We demonstrate that our method improves the accumulated MOTA score for all test sequences by 24.8% over existing state-of-the-art methods. In several ablations on synthetic and real-world sequences, we show that our graph-based, fully end-to-end-learnable approach yields a significant boost in tracking performance.