4DTAM: Non-Rigid Tracking and Mapping via Dynamic Surface Gaussians

TL;DR

This paper introduces 4DTAM, a novel 4D SLAM method that jointly performs non-rigid surface reconstruction and camera tracking using differentiable rendering, Gaussian surface primitives, and a warp-field modeled by an MLP, supported by a new synthetic dataset.

Contribution

It presents the first 4D SLAM approach combining non-rigid tracking and mapping with a new surface representation, deformation modeling, and evaluation dataset.

Findings

Achieves accurate surface reconstruction from monocular streams.

Effectively models complex non-rigid deformations.

Provides a new synthetic dataset for 4D SLAM evaluation.

Abstract

We propose the first 4D tracking and mapping method that jointly performs camera localization and non-rigid surface reconstruction via differentiable rendering. Our approach captures 4D scenes from an online stream of color images with depth measurements or predictions by jointly optimizing scene geometry, appearance, dynamics, and camera ego-motion. Although natural environments exhibit complex non-rigid motions, 4D-SLAM remains relatively underexplored due to its inherent challenges; even with 2.5D signals, the problem is ill-posed because of the high dimensionality of the optimization space. To overcome these challenges, we first introduce a SLAM method based on Gaussian surface primitives that leverages depth signals more effectively than 3D Gaussians, thereby achieving accurate surface reconstruction. To further model non-rigid deformations, we employ a warp-field represented by a multi-layer perceptron (MLP) and introduce a novel camera pose estimation technique along with surface regularization terms that facilitate spatio-temporal reconstruction. In addition to these algorithmic challenges, a significant hurdle in 4D SLAM research is the lack of reliable ground truth and evaluation protocols, primarily due to the difficulty of 4D capture using commodity sensors. To address this, we present a novel open synthetic dataset of everyday objects with diverse motions, leveraging large-scale object models and animation modeling. In summary, we open up the modern 4D-SLAM research by introducing a novel method and evaluation protocols grounded in modern vision and rendering techniques.