Learning to Stylize Novel Views

TL;DR

This paper introduces a point cloud-based method for consistent 3D scene stylization that synthesizes stylized novel views from arbitrary angles, ensuring view consistency and style transfer quality.

Contribution

We propose a novel point cloud aggregation approach that enables consistent stylization of 3D scenes and novel view synthesis from multiple images and a style reference.

Findings

Our method produces more view-consistent stylized images than existing approaches.

Experimental results demonstrate high-quality stylization across diverse real-world scenes.

The approach effectively combines style transfer with 3D scene understanding.

Abstract

We tackle a 3D scene stylization problem - generating stylized images of a scene from arbitrary novel views given a set of images of the same scene and a reference image of the desired style as inputs. Direct solution of combining novel view synthesis and stylization approaches lead to results that are blurry or not consistent across different views. We propose a point cloud-based method for consistent 3D scene stylization. First, we construct the point cloud by back-projecting the image features to the 3D space. Second, we develop point cloud aggregation modules to gather the style information of the 3D scene, and then modulate the features in the point cloud with a linear transformation matrix. Finally, we project the transformed features to 2D space to obtain the novel views. Experimental results on two diverse datasets of real-world scenes validate that our method generates consistent stylized novel view synthesis results against other alternative approaches.