Convolutional Neural Opacity Radiance Fields

TL;DR

This paper introduces a novel convolutional neural radiance field framework that effectively models and renders fuzzy objects with complex opacity, achieving high-quality, view-consistent appearance and opacity in novel views.

Contribution

It combines explicit opacity supervision with convolutional mechanisms in neural radiance fields, enabling detailed and consistent rendering of fuzzy objects from arbitrary viewpoints.

Findings

Achieves photo-realistic rendering of fuzzy objects

Ensures global consistency in appearance and opacity

Demonstrates superior results on a new challenging dataset

Abstract

Photo-realistic modeling and rendering of fuzzy objects with complex opacity are critical for numerous immersive VR/AR applications, but it suffers from strong view-dependent brightness, color. In this paper, we propose a novel scheme to generate opacity radiance fields with a convolutional neural renderer for fuzzy objects, which is the first to combine both explicit opacity supervision and convolutional mechanism into the neural radiance field framework so as to enable high-quality appearance and global consistent alpha mattes generation in arbitrary novel views. More specifically, we propose an efficient sampling strategy along with both the camera rays and image plane, which enables efficient radiance field sampling and learning in a patch-wise manner, as well as a novel volumetric feature integration scheme that generates per-patch hybrid feature embeddings to reconstruct the view-consistent fine-detailed appearance and opacity output. We further adopt a patch-wise adversarial training scheme to preserve both high-frequency appearance and opacity details in a self-supervised framework. We also introduce an effective multi-view image capture system to capture high-quality color and alpha maps for challenging fuzzy objects. Extensive experiments on existing and our new challenging fuzzy object dataset demonstrate that our method achieves photo-realistic, globally consistent, and fined detailed appearance and opacity free-viewpoint rendering for various fuzzy objects.