NeRFiller: Completing Scenes via Generative 3D Inpainting

TL;DR

NeRFiller introduces a novel method for completing missing parts of 3D scenes by leveraging 2D generative models, achieving highly consistent and plausible scene reconstructions without requiring detailed masks or text prompts.

Contribution

It presents a new framework that uses 2D inpainting diffusion models for 3D scene completion, generalizing behavior to multiple images and producing more consistent results.

Findings

NeRFiller outperforms baselines in 3D consistency and plausibility.

The method effectively completes scenes without requiring object masks.

It generalizes 2D inpainting behavior to multiple images for better 3D inpainting.

Abstract

We propose NeRFiller, an approach that completes missing portions of a 3D capture via generative 3D inpainting using off-the-shelf 2D visual generative models. Often parts of a captured 3D scene or object are missing due to mesh reconstruction failures or a lack of observations (e.g., contact regions, such as the bottom of objects, or hard-to-reach areas). We approach this challenging 3D inpainting problem by leveraging a 2D inpainting diffusion model. We identify a surprising behavior of these models, where they generate more 3D consistent inpaints when images form a 2$\times$2 grid, and show how to generalize this behavior to more than four images. We then present an iterative framework to distill these inpainted regions into a single consistent 3D scene. In contrast to related works, we focus on completing scenes rather than deleting foreground objects, and our approach does not require tight 2D object masks or text. We compare our approach to relevant baselines adapted to our setting on a variety of scenes, where NeRFiller creates the most 3D consistent and plausible scene completions. Our project page is at https://ethanweber.me/nerfiller.