Locally Stylized Neural Radiance Fields

TL;DR

This paper introduces a local style transfer method for neural radiance fields that uses hash-grid encoding and a novel loss function to achieve view-consistent stylization with controllable region correspondence.

Contribution

It presents a new framework combining hash-grid encoding and a segmentation-based loss for localized style transfer on NeRFs, enabling flexible and view-consistent stylization.

Findings

Produces plausible stylization with novel view synthesis

Allows controllable and customizable region-based stylization

Maintains appearance consistency across views

Abstract

In recent years, there has been increasing interest in applying stylization on 3D scenes from a reference style image, in particular onto neural radiance fields (NeRF). While performing stylization directly on NeRF guarantees appearance consistency over arbitrary novel views, it is a challenging problem to guide the transfer of patterns from the style image onto different parts of the NeRF scene. In this work, we propose a stylization framework for NeRF based on local style transfer. In particular, we use a hash-grid encoding to learn the embedding of the appearance and geometry components, and show that the mapping defined by the hash table allows us to control the stylization to a certain extent. Stylization is then achieved by optimizing the appearance branch while keeping the geometry branch fixed. To support local style transfer, we propose a new loss function that utilizes a segmentation network and bipartite matching to establish region correspondences between the style image and the content images obtained from volume rendering. Our experiments show that our method yields plausible stylization results with novel view synthesis while having flexible controllability via manipulating and customizing the region correspondences.