View-consistent Object Removal in Radiance Fields

TL;DR

This paper introduces a novel radiance field editing method that ensures view consistency by inpainting a single reference image and projecting it across multiple views, accommodating real-world lighting variations for improved realism.

Contribution

The work proposes a new RF editing pipeline that reduces the need for per-frame inpainting, using depth-based projection and appearance adjustment to maintain consistency across views.

Findings

Significantly improves view consistency in RF editing.

Effectively handles photometric variations across views.

Outperforms existing methods in visual quality and coherence.

Abstract

Radiance Fields (RFs) have emerged as a crucial technology for 3D scene representation, enabling the synthesis of novel views with remarkable realism. However, as RFs become more widely used, the need for effective editing techniques that maintain coherence across different perspectives becomes evident. Current methods primarily depend on per-frame 2D image inpainting, which often fails to maintain consistency across views, thus compromising the realism of edited RF scenes. In this work, we introduce a novel RF editing pipeline that significantly enhances consistency by requiring the inpainting of only a single reference image. This image is then projected across multiple views using a depth-based approach, effectively reducing the inconsistencies observed with per-frame inpainting. However, projections typically assume photometric consistency across views, which is often impractical in real-world settings. To accommodate realistic variations in lighting and viewpoint, our pipeline adjusts the appearance of the projected views by generating multiple directional variants of the inpainted image, thereby adapting to different photometric conditions. Additionally, we present an effective and robust multi-view object segmentation approach as a valuable byproduct of our pipeline. Extensive experiments demonstrate that our method significantly surpasses existing frameworks in maintaining content consistency across views and enhancing visual quality. More results are available at https://vulab-ai.github.io/View-consistent_Object_Removal_in_Radiance_Fields.