StyleMesh: Style Transfer for Indoor 3D Scene Reconstructions

TL;DR

This paper introduces StyleMesh, a method for applying style transfer to 3D indoor scene meshes by optimizing explicit textures with depth and normal awareness, enabling consistent, detailed stylization suitable for real-time rendering.

Contribution

We propose a novel 3D style transfer approach that optimizes explicit textures on meshes using depth and surface normal data, ensuring uniform stylization across the entire scene.

Findings

Produces sharp, detailed stylized scenes without view-dependent artifacts

Enables real-time rendering of stylized meshes in traditional pipelines

Demonstrates effectiveness through extensive ablation studies

Abstract

We apply style transfer on mesh reconstructions of indoor scenes. This enables VR applications like experiencing 3D environments painted in the style of a favorite artist. Style transfer typically operates on 2D images, making stylization of a mesh challenging. When optimized over a variety of poses, stylization patterns become stretched out and inconsistent in size. On the other hand, model-based 3D style transfer methods exist that allow stylization from a sparse set of images, but they require a network at inference time. To this end, we optimize an explicit texture for the reconstructed mesh of a scene and stylize it jointly from all available input images. Our depth- and angle-aware optimization leverages surface normal and depth data of the underlying mesh to create a uniform and consistent stylization for the whole scene. Our experiments show that our method creates sharp and detailed results for the complete scene without view-dependent artifacts. Through extensive ablation studies, we show that the proposed 3D awareness enables style transfer to be applied to the 3D domain of a mesh. Our method can be used to render a stylized mesh in real-time with traditional rendering pipelines.