M2StyleGS: Multi-Modality 3D Style Transfer with Gaussian Splatting

TL;DR

M2StyleGS is a real-time multi-modality 3D style transfer method using Gaussian Splatting and CLIP, enabling flexible reference inputs like text and images for consistent style transfer in 3D scenes.

Contribution

The paper introduces M2StyleGS, a novel 3D style transfer technique that employs Gaussian Splatting and multi-modality knowledge to improve flexibility and consistency.

Findings

Achieves up to 32.92% better consistency than previous methods.

Generates style-enhanced views from text or images as references.

Employs novel loss functions for better style matching and color suppression.

Abstract

Conventional 3D style transfer methods rely on a fixed reference image to apply artistic patterns to 3D scenes. However, in practical applications such as virtual or augmented reality, users often prefer more flexible inputs, including textual descriptions and diverse imagery. In this work, we introduce a novel real-time styling technique M2StyleGS to generate a sequence of precisely color-mapped views. It utilizes 3D Gaussian Splatting (3DGS) as a 3D presentation and multi-modality knowledge refined by CLIP as a reference style. M2StyleGS resolves the abnormal transformation issue by employing a precise feature alignment, namely subdivisive flow, it strengthens the projection of the mapped CLIP text-visual combination feature to the VGG style feature. In addition, we introduce observation loss, which assists in the stylized scene better matching the reference style during the generation, and suppression loss, which suppresses the offset of reference color information throughout the decoding process. By integrating these approaches, M2StyleGS can employ text or images as references to generate a set of style-enhanced novel views. Our experiments show that M2StyleGS achieves better visual quality and surpasses the previous work by up to 32.92% in terms of consistency.