Point Resampling and Ray Transformation Aid to Editable NeRF Models

TL;DR

This paper introduces PR^2T-NeRF, a novel pipeline combining point resampling and ray transformation to improve 3D object editing, removal, and inpainting in NeRF models, achieving state-of-the-art results.

Contribution

It proposes a new implicit ray transformation strategy and a differentiable neural-point resampling module for enhanced object editing and inpainting in NeRFs.

Findings

Achieves state-of-the-art performance in object removal and inpainting tasks.

Supports high-quality rendering for diverse editing operations.

Effectively narrows the gap between ground truth and predicted features.

Abstract

In NeRF-aided editing tasks, object movement presents difficulties in supervision generation due to the introduction of variability in object positions. Moreover, the removal operations of certain scene objects often lead to empty regions, presenting challenges for NeRF models in inpainting them effectively. We propose an implicit ray transformation strategy, allowing for direct manipulation of the 3D object's pose by operating on the neural-point in NeRF rays. To address the challenge of inpainting potential empty regions, we present a plug-and-play inpainting module, dubbed differentiable neural-point resampling (DNR), which interpolates those regions in 3D space at the original ray locations within the implicit space, thereby facilitating object removal & scene inpainting tasks. Importantly, employing DNR effectively narrows the gap between ground truth and predicted implicit features, potentially increasing the mutual information (MI) of the features across rays. Then, we leverage DNR and ray transformation to construct a point-based editable NeRF pipeline PR^2T-NeRF. Results primarily evaluated on 3D object removal & inpainting tasks indicate that our pipeline achieves state-of-the-art performance. In addition, our pipeline supports high-quality rendering visualization for diverse editing operations without necessitating extra supervision.