NeRF-Insert: 3D Local Editing with Multimodal Control Signals

TL;DR

NeRF-Insert introduces a flexible 3D scene editing framework that combines multimodal inputs to perform high-quality, consistent local edits by framing the task as an in-painting problem within NeRFs.

Contribution

It is the first to enable multimodal, local 3D scene editing in NeRFs by integrating textual, visual, and mask inputs through an in-painting approach.

Findings

Achieves higher visual quality in edits compared to previous methods.

Maintains stronger 3D consistency with the original scene.

Supports diverse input modalities for flexible editing.

Abstract

We propose NeRF-Insert, a NeRF editing framework that allows users to make high-quality local edits with a flexible level of control. Unlike previous work that relied on image-to-image models, we cast scene editing as an in-painting problem, which encourages the global structure of the scene to be preserved. Moreover, while most existing methods use only textual prompts to condition edits, our framework accepts a combination of inputs of different modalities as reference. More precisely, a user may provide a combination of textual and visual inputs including images, CAD models, and binary image masks for specifying a 3D region. We use generic image generation models to in-paint the scene from multiple viewpoints, and lift the local edits to a 3D-consistent NeRF edit. Compared to previous methods, our results show better visual quality and also maintain stronger consistency with the original NeRF.