DNRSelect: Active Best View Selection for Deferred Neural Rendering

TL;DR

DNRSelect introduces a reinforcement learning-based view selection method and a 3D texture aggregator to reduce reliance on extensive ray-traced images in deferred neural rendering, maintaining high fidelity with fewer resources.

Contribution

It presents a novel view selector trained on rasterized images and a 3D texture aggregator to improve efficiency and geometric consistency in deferred neural rendering.

Findings

Achieves high-quality rendering with fewer ray-traced images.

Effectively integrates rasterized and ray-traced data for better spatial awareness.

Demonstrates superior performance on NeRF-Synthetic dataset.

Abstract

Deferred neural rendering (DNR) is an emerging computer graphics pipeline designed for high-fidelity rendering and robotic perception. However, DNR heavily relies on datasets composed of numerous ray-traced images and demands substantial computational resources. It remains under-explored how to reduce the reliance on high-quality ray-traced images while maintaining the rendering fidelity. In this paper, we propose DNRSelect, which integrates a reinforcement learning-based view selector and a 3D texture aggregator for deferred neural rendering. We first propose a novel view selector for deferred neural rendering based on reinforcement learning, which is trained on easily obtained rasterized images to identify the optimal views. By acquiring only a few ray-traced images for these selected views, the selector enables DNR to achieve high-quality rendering. To further enhance spatial awareness and geometric consistency in DNR, we introduce a 3D texture aggregator that fuses pyramid features from depth maps and normal maps with UV maps. Given that acquiring ray-traced images is more time-consuming than generating rasterized images, DNRSelect minimizes the need for ray-traced data by using only a few selected views while still achieving high-fidelity rendering results. We conduct detailed experiments and ablation studies on the NeRF-Synthetic dataset to demonstrate the effectiveness of DNRSelect. The code will be released.