Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis

TL;DR

DMTet is a novel hybrid 3D shape synthesis model that combines implicit and explicit representations, enabling high-resolution, detailed, and topologically flexible 3D shape generation from coarse inputs.

Contribution

The paper introduces DMTet, a hybrid deep generative model that directly optimizes surface reconstruction and topology, outperforming existing implicit and explicit 3D shape synthesis methods.

Findings

Outperforms existing methods on 3D animal shape synthesis

Enables high-resolution shape details with fewer artifacts

Supports arbitrary topology in generated shapes

Abstract

We introduce DMTet, a deep 3D conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such as coarse voxels. It marries the merits of implicit and explicit 3D representations by leveraging a novel hybrid 3D representation. Compared to the current implicit approaches, which are trained to regress the signed distance values, DMTet directly optimizes for the reconstructed surface, which enables us to synthesize finer geometric details with fewer artifacts. Unlike deep 3D generative models that directly generate explicit representations such as meshes, our model can synthesize shapes with arbitrary topology. The core of DMTet includes a deformable tetrahedral grid that encodes a discretized signed distance function and a differentiable marching tetrahedra layer that converts the implicit signed distance representation to the explicit surface mesh representation. This combination allows joint optimization of the surface geometry and topology as well as generation of the hierarchy of subdivisions using reconstruction and adversarial losses defined explicitly on the surface mesh. Our approach significantly outperforms existing work on conditional shape synthesis from coarse voxel inputs, trained on a dataset of complex 3D animal shapes. Project page: https://nv-tlabs.github.io/DMTet/.