Towards Multi-View Consistent Style Transfer with One-Step Diffusion via Vision Conditioning

TL;DR

This paper introduces OSDiffST, a novel style transfer method for 3D scenes that maintains multi-view consistency and structural integrity using a one-step diffusion model with vision conditioning.

Contribution

The paper proposes a new diffusion-based style transfer approach with a vision conditioning module and LoRA adaptation, specifically designed for multi-view 3D scene stylization.

Findings

Outperforms existing style transfer methods in multi-view consistency.

Produces stylized images with better structural preservation and less distortion.

Effectively adapts pre-trained diffusion models for small datasets.

Abstract

The stylization of 3D scenes is an increasingly attractive topic in 3D vision. Although image style transfer has been extensively researched with promising results, directly applying 2D style transfer methods to 3D scenes often fails to preserve the structural and multi-view properties of 3D environments, resulting in unpleasant distortions in images from different viewpoints. To address these issues, we leverage the remarkable generative prior of diffusion-based models and propose a novel style transfer method, OSDiffST, based on a pre-trained one-step diffusion model (i.e., SD-Turbo) for rendering diverse styles in multi-view images of 3D scenes. To efficiently adapt the pre-trained model for multi-view style transfer on small datasets, we introduce a vision condition module to extract style information from the reference style image to serve as conditional input for the diffusion model and employ LoRA in diffusion model for adaptation. Additionally, we consider color distribution alignment and structural similarity between the stylized and content images using two specific loss functions. As a result, our method effectively preserves the structural information and multi-view consistency in stylized images without any 3D information. Experiments show that our method surpasses other promising style transfer methods in synthesizing various styles for multi-view images of 3D scenes. Stylized images from different viewpoints generated by our method achieve superior visual quality, with better structural integrity and less distortion. The source code is available at https://github.com/YushenZuo/OSDiffST.