Language-driven Object Fusion into Neural Radiance Fields with Pose-Conditioned Dataset Updates

TL;DR

This paper introduces a language-driven method for inserting and removing objects in neural radiance fields by updating datasets with combined multi-view images, enabling photorealistic scene editing and improved 3D reconstruction.

Contribution

It presents a novel dataset update strategy guided by language models to manipulate objects in neural radiance fields, enhancing scene editing capabilities.

Findings

Produces photorealistic scene edits

Outperforms state-of-the-art in 3D reconstruction

Enables flexible object insertion and removal

Abstract

Neural radiance field is an emerging rendering method that generates high-quality multi-view consistent images from a neural scene representation and volume rendering. Although neural radiance field-based techniques are robust for scene reconstruction, their ability to add or remove objects remains limited. This paper proposes a new language-driven approach for object manipulation with neural radiance fields through dataset updates. Specifically, to insert a new foreground object represented by a set of multi-view images into a background radiance field, we use a text-to-image diffusion model to learn and generate combined images that fuse the object of interest into the given background across views. These combined images are then used for refining the background radiance field so that we can render view-consistent images containing both the object and the background. To ensure view consistency, we propose a dataset updates strategy that prioritizes radiance field training with camera views close to the already-trained views prior to propagating the training to remaining views. We show that under the same dataset updates strategy, we can easily adapt our method for object insertion using data from text-to-3D models as well as object removal. Experimental results show that our method generates photorealistic images of the edited scenes, and outperforms state-of-the-art methods in 3D reconstruction and neural radiance field blending.