Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments

TL;DR

This paper presents a novel scene reconstruction method that combines panoptic mapping with CAD model alignment to produce physically plausible, interaction-ready 3D scenes from RGB-D data, enhancing robot interaction capabilities.

Contribution

The work introduces a new approach that integrates semantic, geometric, and physical reasoning with CAD model alignment for interactive scene reconstruction.

Findings

Outperforms previous panoptic mapping methods

Achieves high-accuracy CAD model alignment and replacement

Produces scenes suitable for robot interaction and simulation

Abstract

In this paper, we rethink the problem of scene reconstruction from an embodied agent's perspective: While the classic view focuses on the reconstruction accuracy, our new perspective emphasizes the underlying functions and constraints such that the reconstructed scenes provide \em{actionable} information for simulating \em{interactions} with agents. Here, we address this challenging problem by reconstructing an interactive scene using RGB-D data stream, which captures (i) the semantics and geometry of objects and layouts by a 3D volumetric panoptic mapping module, and (ii) object affordance and contextual relations by reasoning over physical common sense among objects, organized by a graph-based scene representation. Crucially, this reconstructed scene replaces the object meshes in the dense panoptic map with part-based articulated CAD models for finer-grained robot interactions. In the experiments, we demonstrate that (i) our panoptic mapping module outperforms previous state-of-the-art methods, (ii) a high-performant physical reasoning procedure that matches, aligns, and replaces objects' meshes with best-fitted CAD models, and (iii) reconstructed scenes are physically plausible and naturally afford actionable interactions; without any manual labeling, they are seamlessly imported to ROS-based simulators and virtual environments for complex robot task executions.