Memorize What Matters: Emergent Scene Decomposition from Multitraverse

TL;DR

This paper introduces 3D Gaussian Mapping (3DGM), a self-supervised framework that decomposes environments into persistent and ephemeral elements from multitraverse RGB videos, enhancing robotic perception and mapping.

Contribution

The paper presents a novel self-supervised, camera-only 3D mapping method that jointly performs environment-object decomposition and ephemeral object segmentation from multitraverse videos.

Findings

Effective environment-object decomposition demonstrated

Achieved accurate 2D segmentation and 3D reconstruction

Validated on the new Mapverse benchmark

Abstract

Humans naturally retain memories of permanent elements, while ephemeral moments often slip through the cracks of memory. This selective retention is crucial for robotic perception, localization, and mapping. To endow robots with this capability, we introduce 3D Gaussian Mapping (3DGM), a self-supervised, camera-only offline mapping framework grounded in 3D Gaussian Splatting. 3DGM converts multitraverse RGB videos from the same region into a Gaussian-based environmental map while concurrently performing 2D ephemeral object segmentation. Our key observation is that the environment remains consistent across traversals, while objects frequently change. This allows us to exploit self-supervision from repeated traversals to achieve environment-object decomposition. More specifically, 3DGM formulates multitraverse environmental mapping as a robust differentiable rendering problem, treating pixels of the environment and objects as inliers and outliers, respectively. Using robust feature distillation, feature residuals mining, and robust optimization, 3DGM jointly performs 2D segmentation and 3D mapping without human intervention. We build the Mapverse benchmark, sourced from the Ithaca365 and nuPlan datasets, to evaluate our method in unsupervised 2D segmentation, 3D reconstruction, and neural rendering. Extensive results verify the effectiveness and potential of our method for self-driving and robotics.