Learning a Diffusion Prior for NeRFs

TL;DR

This paper introduces a diffusion model-based approach to generate neural radiance fields (NeRFs), improving the ability to produce realistic 3D scene representations from limited views and enabling conditional scene generation.

Contribution

It proposes a novel diffusion prior for NeRFs that enhances generation quality and allows conditional sampling based on observations.

Findings

Generated realistic NeRFs with the diffusion model

Enabled conditional NeRF generation from partial observations

Improved NeRF synthesis in low-data scenarios

Abstract

Neural Radiance Fields (NeRFs) have emerged as a powerful neural 3D representation for objects and scenes derived from 2D data. Generating NeRFs, however, remains difficult in many scenarios. For instance, training a NeRF with only a small number of views as supervision remains challenging since it is an under-constrained problem. In such settings, it calls for some inductive prior to filter out bad local minima. One way to introduce such inductive priors is to learn a generative model for NeRFs modeling a certain class of scenes. In this paper, we propose to use a diffusion model to generate NeRFs encoded on a regularized grid. We show that our model can sample realistic NeRFs, while at the same time allowing conditional generations, given a certain observation as guidance.