IDDR-NGP: Incorporating Detectors for Distractor Removal with Instant Neural Radiance Field

TL;DR

IDDR-NGP introduces a unified method for removing diverse distractors from 3D scenes using implicit representations and multi-view optimization, outperforming existing specialized techniques.

Contribution

The paper presents the first unified distractor removal method for 3D scenes that operates directly on Instant-NGP, incorporating 2D detectors and novel loss functions for robust multi-view scene restoration.

Findings

Effectively removes various distractors including snowflakes, confetti, and petals.

Achieves comparable results to state-of-the-art desnow methods.

Demonstrates robustness on both synthetic and real-world scenes.

Abstract

This paper presents the first unified distractor removal method, named IDDR-NGP, which directly operates on Instant-NPG. The method is able to remove a wide range of distractors in 3D scenes, such as snowflakes, confetti, defoliation and petals, whereas existing methods usually focus on a specific type of distractors. By incorporating implicit 3D representations with 2D detectors, we demonstrate that it is possible to efficiently restore 3D scenes from multiple corrupted images. We design the learned perceptual image patch similarity~( LPIPS) loss and the multi-view compensation loss (MVCL) to jointly optimize the rendering results of IDDR-NGP, which could aggregate information from multi-view corrupted images. All of them can be trained in an end-to-end manner to synthesize high-quality 3D scenes. To support the research on distractors removal in implicit 3D representations, we build a new benchmark dataset that consists of both synthetic and real-world distractors. To validate the effectiveness and robustness of IDDR-NGP, we provide a wide range of distractors with corresponding annotated labels added to both realistic and synthetic scenes. Extensive experimental results demonstrate the effectiveness and robustness of IDDR-NGP in removing multiple types of distractors. In addition, our approach achieves results comparable with the existing SOTA desnow methods and is capable of accurately removing both realistic and synthetic distractors.