SIn-NeRF2NeRF: Editing 3D Scenes with Instructions through Segmentation and Inpainting

TL;DR

This paper introduces a method for editing 3D scenes by disentangling objects from backgrounds using segmentation and inpainting, enabling flexible geometric modifications like resizing and moving within Neural Radiance Fields.

Contribution

It presents a novel approach for selective 3D object editing in NeRF scenes through segmentation and inpainting, allowing geometric modifications after disentangling objects from backgrounds.

Findings

Effective object segmentation and background inpainting in 3D scenes.

Successful resizing and repositioning of objects within NeRF models.

Demonstrated versatility in editing complex 3D scenes.

Abstract

TL;DR Perform 3D object editing selectively by disentangling it from the background scene. Instruct-NeRF2NeRF (in2n) is a promising method that enables editing of 3D scenes composed of Neural Radiance Field (NeRF) using text prompts. However, it is challenging to perform geometrical modifications such as shrinking, scaling, or moving on both the background and object simultaneously. In this project, we enable geometrical changes of objects within the 3D scene by selectively editing the object after separating it from the scene. We perform object segmentation and background inpainting respectively, and demonstrate various examples of freely resizing or moving disentangled objects within the three-dimensional space.