Dream-to-Recon: Monocular 3D Reconstruction with Diffusion-Depth Distillation from Single Images

TL;DR

This paper introduces a novel method for monocular 3D scene reconstruction that uses diffusion and depth models to generate synthetic geometry, enabling high-quality reconstruction without multi-view supervision.

Contribution

It presents a new approach combining diffusion models and depth prediction for single-image 3D reconstruction, reducing reliance on expensive ground truth data.

Findings

Outperforms multi-view supervised methods on KITTI-360 and Waymo datasets

Effective in dynamic scene reconstruction

Matches or exceeds state-of-the-art accuracy

Abstract

Volumetric scene reconstruction from a single image is crucial for a broad range of applications like autonomous driving and robotics. Recent volumetric reconstruction methods achieve impressive results, but generally require expensive 3D ground truth or multi-view supervision. We propose to leverage pre-trained 2D diffusion models and depth prediction models to generate synthetic scene geometry from a single image. This can then be used to distill a feed-forward scene reconstruction model. Our experiments on the challenging KITTI-360 and Waymo datasets demonstrate that our method matches or outperforms state-of-the-art baselines that use multi-view supervision, and offers unique advantages, for example regarding dynamic scenes.