Object-centric Reconstruction and Tracking of Dynamic Unknown Objects using 3D Gaussian Splatting

TL;DR

This paper introduces a novel, online method for reconstructing and tracking unknown, dynamic objects in 3D space using Gaussian blobs, without prior knowledge or pre-training, suitable for space robotics applications.

Contribution

It adapts 3D Gaussian Splatting to a dynamic, object-centric setting for the first time, enabling real-time reconstruction and tracking of unknown objects from RGB-D data.

Findings

Successful 3D reconstruction of diverse spacecraft objects

Accurate 6-DoF pose tracking in dynamic scenarios

No pre-training or prior object knowledge required

Abstract

Generalizable perception is one of the pillars of high-level autonomy in space robotics. Estimating the structure and motion of unknown objects in dynamic environments is fundamental for such autonomous systems. Traditionally, the solutions have relied on prior knowledge of target objects, multiple disparate representations, or low-fidelity outputs unsuitable for robotic operations. This work proposes a novel approach to incrementally reconstruct and track a dynamic unknown object using a unified representation -- a set of 3D Gaussian blobs that describe its geometry and appearance. The differentiable 3D Gaussian Splatting framework is adapted to a dynamic object-centric setting. The input to the pipeline is a sequential set of RGB-D images. 3D reconstruction and 6-DoF pose tracking tasks are tackled using first-order gradient-based optimization. The formulation is simple, requires no pre-training, assumes no prior knowledge of the object or its motion, and is suitable for online applications. The proposed approach is validated on a dataset of 10 unknown spacecraft of diverse geometry and texture under arbitrary relative motion. The experiments demonstrate successful 3D reconstruction and accurate 6-DoF tracking of the target object in proximity operations over a short to medium duration. The causes of tracking drift are discussed and potential solutions are outlined.