Rescan: Inductive Instance Segmentation for Indoor RGBD Scans

TL;DR

Rescan introduces an inductive method for temporal scene modeling and instance segmentation in indoor RGBD scans, enabling object tracking over time with sparse observations.

Contribution

It presents a novel inductive algorithm that leverages past scans to improve instance segmentation and object tracking in indoor 3D scene reconstructions.

Findings

Outperforms state-of-the-art networks on a new benchmark

Effective in tracking objects over time with sparse data

Provides a temporal model for scene understanding

Abstract

In depth-sensing applications ranging from home robotics to AR/VR, it will be common to acquire 3D scans of interior spaces repeatedly at sparse time intervals (e.g., as part of regular daily use). We propose an algorithm that analyzes these "rescans" to infer a temporal model of a scene with semantic instance information. Our algorithm operates inductively by using the temporal model resulting from past observations to infer an instance segmentation of a new scan, which is then used to update the temporal model. The model contains object instance associations across time and thus can be used to track individual objects, even though there are only sparse observations. During experiments with a new benchmark for the new task, our algorithm outperforms alternate approaches based on state-of-the-art networks for semantic instance segmentation.