Light Transport-aware Diffusion Posterior Sampling for Single-View Reconstruction of 3D Volumes

TL;DR

This paper presents a novel single-view volumetric reconstruction method that leverages a diffusion model trained on synthetic data and a differentiable volume renderer to improve the quality of 3D cloud reconstructions.

Contribution

It introduces a diffusion-based generative model trained on a new dataset and a diffusion-aware posterior sampling technique for enhanced single-view 3D volume reconstruction.

Findings

Achieved high-quality single-view cloud reconstructions.

Demonstrated superiority over classic NeRF approaches.

Validated effectiveness on synthetic volumetric cloud data.

Abstract

We introduce a single-view reconstruction technique of volumetric fields in which multiple light scattering effects are omnipresent, such as in clouds. We model the unknown distribution of volumetric fields using an unconditional diffusion model trained on a novel benchmark dataset comprising 1,000 synthetically simulated volumetric density fields. The neural diffusion model is trained on the latent codes of a novel, diffusion-friendly, monoplanar representation. The generative model is used to incorporate a tailored parametric diffusion posterior sampling technique into different reconstruction tasks. A physically-based differentiable volume renderer is employed to provide gradients with respect to light transport in the latent space. This stands in contrast to classic NeRF approaches and makes the reconstructions better aligned with observed data. Through various experiments, we demonstrate single-view reconstruction of volumetric clouds at a previously unattainable quality.