O$^2$-Recon: Completing 3D Reconstruction of Occluded Objects in the Scene with a Pre-trained 2D Diffusion Model

TL;DR

This paper introduces O$^2$-Recon, a novel framework that leverages a pre-trained 2D diffusion model and a cascaded network architecture to improve 3D reconstruction of occluded objects from RGB-D videos, achieving state-of-the-art results.

Contribution

The paper presents a new method combining 2D diffusion in-painting, minimal human-in-the-loop mask generation, and a cascaded network for predicting signed distance fields, enhancing 3D object reconstruction.

Findings

Achieves state-of-the-art accuracy in object reconstruction.

Effectively reconstructs occluded and hidden object parts.

Utilizes semantic consistency loss for better unseen view reconstruction.

Abstract

Occlusion is a common issue in 3D reconstruction from RGB-D videos, often blocking the complete reconstruction of objects and presenting an ongoing problem. In this paper, we propose a novel framework, empowered by a 2D diffusion-based in-painting model, to reconstruct complete surfaces for the hidden parts of objects. Specifically, we utilize a pre-trained diffusion model to fill in the hidden areas of 2D images. Then we use these in-painted images to optimize a neural implicit surface representation for each instance for 3D reconstruction. Since creating the in-painting masks needed for this process is tricky, we adopt a human-in-the-loop strategy that involves very little human engagement to generate high-quality masks. Moreover, some parts of objects can be totally hidden because the videos are usually shot from limited perspectives. To ensure recovering these invisible areas, we develop a cascaded network architecture for predicting signed distance field, making use of different frequency bands of positional encoding and maintaining overall smoothness. Besides the commonly used rendering loss, Eikonal loss, and silhouette loss, we adopt a CLIP-based semantic consistency loss to guide the surface from unseen camera angles. Experiments on ScanNet scenes show that our proposed framework achieves state-of-the-art accuracy and completeness in object-level reconstruction from scene-level RGB-D videos. Code: https://github.com/THU-LYJ-Lab/O2-Recon.