UniMGS: Unifying Mesh and 3D Gaussian Splatting with Single-Pass Rasterization and Proxy-Based Deformation

TL;DR

UniMGS introduces a unified, single-pass rasterization framework that seamlessly combines mesh and 3D Gaussian Splatting rendering, improving visual coherence and handling occlusions and transparency more accurately.

Contribution

The paper presents the first unified rasterization framework for mesh and 3D Gaussian Splatting, enabling joint rendering and deformation in a single pass with novel blending and binding strategies.

Findings

Achieves visually coherent rendering of mesh and 3DGS in a single pass.

Reduces artifacts in deformed 3DGS using proxy mesh-based Gaussian binding.

Enables unified visualization and manipulation of mesh and 3DGS objects.

Abstract

Joint rendering and deformation of mesh and 3D Gaussian Splatting (3DGS) have significant value as both representa tions offer complementary advantages for graphics applica tions. However, due to differences in representation and ren dering pipelines, existing studies render meshes and 3DGS separately, making it difficult to accurately handle occlusions and transparency. Moreover, the deformed 3DGS still suffers from visual artifacts due to the sensitivity to the topology quality of the proxy mesh. These issues pose serious obsta cles to the joint use of 3DGS and meshes, making it diffi cult to adapt 3DGS to conventional mesh-oriented graphics pipelines. We propose UniMGS, the first unified framework for rasterizing mesh and 3DGS in a single-pass anti-aliased manner, with a novel binding strategy for 3DGS deformation based on proxy mesh. Our key insight is to blend the col ors of both triangle and Gaussian fragments by anti-aliased {\alpha}-blending in a single pass, achieving visually coherent re sults with precise handling of occlusion and transparency. To improve the visual appearance of the deformed 3DGS, our Gaussian-centric binding strategy employs a proxy mesh and spatially associates Gaussians with the mesh faces, signifi cantly reducing rendering artifacts. With these two compo nents, UniMGS enables the visualization and manipulation of 3D objects represented by mesh or 3DGS within a unified framework, opening up new possibilities in embodied AI, vir tual reality, and gaming. We will release our source code to facilitate future research.