Enhancement of Novel View Synthesis Using Omnidirectional Image Completion

TL;DR

This paper introduces a novel approach for synthesizing new views from a single 360-degree RGB-D image by completing missing regions with a 2D generative model and training NeRF with selected consistent images.

Contribution

It proposes a new method combining image completion and optimized image selection to improve novel view synthesis from a single omnidirectional image.

Findings

Effective in synthesizing plausible novel views

Preserves scene features in both artificial and real-world data

Reduces artifacts caused by occlusion and zooming

Abstract

In this study, we present a method for synthesizing novel views from a single 360-degree RGB-D image based on the neural radiance field (NeRF) . Prior studies relied on the neighborhood interpolation capability of multi-layer perceptrons to complete missing regions caused by occlusion and zooming, which leads to artifacts. In the method proposed in this study, the input image is reprojected to 360-degree RGB images at other camera positions, the missing regions of the reprojected images are completed by a 2D image generative model, and the completed images are utilized to train the NeRF. Because multiple completed images contain inconsistencies in 3D, we introduce a method to learn the NeRF model using a subset of completed images that cover the target scene with less overlap of completed regions. The selection of such a subset of images can be attributed to the maximum weight independent set problem, which is solved through simulated annealing. Experiments demonstrated that the proposed method can synthesize plausible novel views while preserving the features of the scene for both artificial and real-world data.