Multi-StyleGS: Stylizing Gaussian Splatting with Multiple Styles

TL;DR

Multi-StyleGS introduces a novel method for stylizing 3D Gaussian Splatting scenes with multiple styles, employing a bipartite matching mechanism and semantic style loss to achieve memory-efficient, multi-view consistent, and detailed stylizations.

Contribution

It presents a new 3D stylization approach that enables automatic multi-style transfer with semantic segmentation and local-global feature matching, improving over prior methods.

Findings

Outperforms existing methods in stylization quality

Achieves better multi-view consistency and detail

Enables flexible editing of 3D scenes

Abstract

In recent years, there has been a growing demand to stylize a given 3D scene to align with the artistic style of reference images for creative purposes. While 3D Gaussian Splatting(GS) has emerged as a promising and efficient method for realistic 3D scene modeling, there remains a challenge in adapting it to stylize 3D GS to match with multiple styles through automatic local style transfer or manual designation, while maintaining memory efficiency for stylization training. In this paper, we introduce a novel 3D GS stylization solution termed Multi-StyleGS to tackle these challenges. In particular, we employ a bipartite matching mechanism to au tomatically identify correspondences between the style images and the local regions of the rendered images. To facilitate local style transfer, we introduce a novel semantic style loss function that employs a segmentation network to apply distinct styles to various objects of the scene and propose a local-global feature matching to enhance the multi-view consistency. Furthermore, this technique can achieve memory efficient training, more texture details and better color match. To better assign a robust semantic label to each Gaussian, we propose several techniques to regularize the segmentation network. As demonstrated by our comprehensive experiments, our approach outperforms existing ones in producing plausible stylization results and offering flexible editing.