Edit-DiffNeRF: Editing 3D Neural Radiance Fields using 2D Diffusion Model

TL;DR

Edit-DiffNeRF introduces a novel framework combining frozen diffusion models, a delta editing module, and NeRF to enable fine-grained, semantically consistent 3D scene editing guided by text instructions, improving alignment accuracy.

Contribution

The paper proposes a new method for editing 3D scenes using a frozen diffusion model and a delta module, addressing cross-view inconsistency and enabling precise semantic modifications.

Findings

Achieves 25% improvement in edit-text alignment accuracy

Effectively edits real-world 3D scenes

Ensures multi-view semantic consistency

Abstract

Recent research has demonstrated that the combination of pretrained diffusion models with neural radiance fields (NeRFs) has emerged as a promising approach for text-to-3D generation. Simply coupling NeRF with diffusion models will result in cross-view inconsistency and degradation of stylized view syntheses. To address this challenge, we propose the Edit-DiffNeRF framework, which is composed of a frozen diffusion model, a proposed delta module to edit the latent semantic space of the diffusion model, and a NeRF. Instead of training the entire diffusion for each scene, our method focuses on editing the latent semantic space in frozen pretrained diffusion models by the delta module. This fundamental change to the standard diffusion framework enables us to make fine-grained modifications to the rendered views and effectively consolidate these instructions in a 3D scene via NeRF training. As a result, we are able to produce an edited 3D scene that faithfully aligns to input text instructions. Furthermore, to ensure semantic consistency across different viewpoints, we propose a novel multi-view semantic consistency loss that extracts a latent semantic embedding from the input view as a prior, and aim to reconstruct it in different views. Our proposed method has been shown to effectively edit real-world 3D scenes, resulting in 25% improvement in the alignment of the performed 3D edits with text instructions compared to prior work.