ReScene4D: Temporally Consistent Semantic Instance Segmentation of Evolving Indoor 3D Scenes

TL;DR

ReScene4D introduces a novel approach for temporally consistent semantic instance segmentation in evolving indoor 3D scenes, enabling better tracking and understanding of dynamic environments over time.

Contribution

It formalizes the task of 4D indoor semantic instance segmentation and proposes ReScene4D, a method that leverages temporal information sharing without dense observations.

Findings

ReScene4D achieves state-of-the-art results on the 3RScan dataset.

The new t-mAP metric effectively measures temporal identity consistency.

ReScene4D improves existing 3DSIS methods by incorporating temporal reasoning.

Abstract

Indoor environments evolve as objects move, appear, or leave the scene. Capturing these dynamics requires maintaining temporally consistent instance identities across intermittently captured 3D scans, even when changes are unobserved. We introduce and formalize the task of temporally sparse 4D indoor semantic instance segmentation (SIS), which jointly segments, identifies, and temporally associates object instances. This setting poses a challenge for existing 3DSIS methods, which require a discrete matching step due to their lack of temporal reasoning, and for 4D LiDAR approaches, which perform poorly due to their reliance on high-frequency temporal measurements that are uncommon in the longer-horizon evolution of indoor environments. We propose ReScene4D, a novel method that adapts 3DSIS architectures for 4DSIS without needing dense observations. Our method enables temporal information sharing--using spatiotemporal contrastive loss, masking, and serialization--to adaptively leverage geometric and semantic priors across observations. This shared context enables consistent instance tracking and improves standard 3DSIS performance. To evaluate this task, we define a new metric, t-mAP, that extends mAP to reward temporal identity consistency. ReScene4D achieves state-of-the-art performance on the 3RScan dataset, establishing a new benchmark for understanding evolving indoor scenes.