Inpaint4DNeRF: Promptable Spatio-Temporal NeRF Inpainting with Generative Diffusion Models

TL;DR

Inpaint4DNeRF introduces a novel method combining NeRF with generative diffusion models to enable photorealistic inpainting of 3D scenes, maintaining multiview and temporal consistency for static and dynamic environments.

Contribution

The paper presents a new generative NeRF inpainting framework leveraging diffusion models for plausible content generation and multiview consistency, extendable to dynamic 4D scenes.

Findings

Effective inpainting of occluded content in 3D scenes.

Maintains multiview and temporal consistency.

Compatible with static and dynamic NeRFs.

Abstract

Current Neural Radiance Fields (NeRF) can generate photorealistic novel views. For editing 3D scenes represented by NeRF, with the advent of generative models, this paper proposes Inpaint4DNeRF to capitalize on state-of-the-art stable diffusion models (e.g., ControlNet) for direct generation of the underlying completed background content, regardless of static or dynamic. The key advantages of this generative approach for NeRF inpainting are twofold. First, after rough mask propagation, to complete or fill in previously occluded content, we can individually generate a small subset of completed images with plausible content, called seed images, from which simple 3D geometry proxies can be derived. Second and the remaining problem is thus 3D multiview consistency among all completed images, now guided by the seed images and their 3D proxies. Without other bells and whistles, our generative Inpaint4DNeRF baseline framework is general which can be readily extended to 4D dynamic NeRFs, where temporal consistency can be naturally handled in a similar way as our multiview consistency.