MessyKitchens: Contact-rich object-level 3D scene reconstruction

TL;DR

This paper introduces MessyKitchens, a new dataset and method for physically plausible 3D scene reconstruction at the object level, improving accuracy and contact modeling in cluttered environments.

Contribution

The paper presents a new dataset with high-fidelity object-level ground truth and extends the SAM 3D approach with Multi-Object Decoder for improved scene reconstruction.

Findings

MessyKitchens dataset outperforms previous datasets in registration accuracy.

Multi-Object Decoder (MOD) significantly improves multi-object scene reconstruction.

Approach demonstrates consistent improvements across multiple datasets.

Abstract

Monocular 3D scene reconstruction has recently seen significant progress. Powered by the modern neural architectures and large-scale data, recent methods achieve high performance in depth estimation from a single image. Meanwhile, reconstructing and decomposing common scenes into individual 3D objects remains a hard challenge due to the large variety of objects, frequent occlusions and complex object relations. Notably, beyond shape and pose estimation of individual objects, applications in robotics and animation require physically-plausible scene reconstruction where objects obey physical principles of non-penetration and realistic contacts. In this work we advance object-level scene reconstruction along two directions. First, we introduceMessyKitchens, a new dataset with real-world scenes featuring cluttered environments and providing high-fidelity object-level ground truth in terms of 3D object shapes, poses and accurate object contacts. Second, we build on the recent SAM 3D approach for single-object reconstruction and extend it with Multi-Object Decoder (MOD) for joint object-level scene reconstruction. To validate our contributions, we demonstrate MessyKitchens to significantly improve previous datasets in registration accuracy and inter-object penetration. We also compare our multi-object reconstruction approach on three datasets and demonstrate consistent and significant improvements of MOD over the state of the art. Our new benchmark, code and pre-trained models will become publicly available on our project website: https://messykitchens.github.io/.