Ref-NPR: Reference-Based Non-Photorealistic Radiance Fields for Controllable Scene Stylization

TL;DR

Ref-NPR introduces a controllable 3D scene stylization method that leverages a single stylized reference view and semantic correspondences to produce non-photorealistic, semantically consistent novel views, outperforming existing methods.

Contribution

It proposes a novel reference-based radiance field approach for controllable scene stylization using a single reference view and semantic matching.

Findings

Outperforms existing methods in visual quality and semantic correspondence

Produces continuous, non-photorealistic novel view sequences

Utilizes pseudo-ray supervision for novel view synthesis

Abstract

Current 3D scene stylization methods transfer textures and colors as styles using arbitrary style references, lacking meaningful semantic correspondences. We introduce Reference-Based Non-Photorealistic Radiance Fields (Ref-NPR) to address this limitation. This controllable method stylizes a 3D scene using radiance fields with a single stylized 2D view as a reference. We propose a ray registration process based on the stylized reference view to obtain pseudo-ray supervision in novel views. Then we exploit semantic correspondences in content images to fill occluded regions with perceptually similar styles, resulting in non-photorealistic and continuous novel view sequences. Our experimental results demonstrate that Ref-NPR outperforms existing scene and video stylization methods regarding visual quality and semantic correspondence. The code and data are publicly available on the project page at https://ref-npr.github.io.