Tetrahedron Splatting for 3D Generation

TL;DR

This paper introduces Tetrahedron Splatting, a novel 3D representation that enables fast convergence, precise mesh extraction, and real-time rendering, improving upon existing methods in 3D generation tasks.

Contribution

The paper presents Tetrahedron Splatting, a new 3D representation integrating surface-based volumetric rendering with tetrahedral grids, supporting efficient optimization and high-quality mesh extraction.

Findings

Achieves faster convergence and better mesh quality than previous methods.

Supports real-time rendering with high efficiency.

Improves 3D generation quality and stability through regularization.

Abstract

3D representation is essential to the significant advance of 3D generation with 2D diffusion priors. As a flexible representation, NeRF has been first adopted for 3D representation. With density-based volumetric rendering, it however suffers both intensive computational overhead and inaccurate mesh extraction. Using a signed distance field and Marching Tetrahedra, DMTet allows for precise mesh extraction and real-time rendering but is limited in handling large topological changes in meshes, leading to optimization challenges. Alternatively, 3D Gaussian Splatting (3DGS) is favored in both training and rendering efficiency while falling short in mesh extraction. In this work, we introduce a novel 3D representation, Tetrahedron Splatting (TeT-Splatting), that supports easy convergence during optimization, precise mesh extraction, and real-time rendering simultaneously. This is achieved by integrating surface-based volumetric rendering within a structured tetrahedral grid while preserving the desired ability of precise mesh extraction, and a tile-based differentiable tetrahedron rasterizer. Furthermore, we incorporate eikonal and normal consistency regularization terms for the signed distance field to improve generation quality and stability. Critically, our representation can be trained without mesh extraction, making the optimization process easier to converge. Our TeT-Splatting can be readily integrated in existing 3D generation pipelines, along with polygonal mesh for texture optimization. Extensive experiments show that our TeT-Splatting strikes a superior tradeoff among convergence speed, render efficiency, and mesh quality as compared to previous alternatives under varying 3D generation settings.