DeepMesh: Differentiable Iso-Surface Extraction

TL;DR

DeepMesh introduces a novel differentiable method to extract explicit mesh surfaces from deep implicit fields, enabling end-to-end training and improved performance in 3D reconstruction and shape optimization.

Contribution

It presents the first differentiable approach to convert implicit fields into explicit meshes, allowing for topology variation and integration into deep learning pipelines.

Findings

Outperforms state-of-the-art in 3D reconstruction tasks

Enables end-to-end training with differentiable rendering

Supports topology changes in mesh extraction

Abstract

Geometric Deep Learning has recently made striking progress with the advent of continuous deep implicit fields. They allow for detailed modeling of watertight surfaces of arbitrary topology while not relying on a 3D Euclidean grid, resulting in a learnable parameterization that is unlimited in resolution. Unfortunately, these methods are often unsuitable for applications that require an explicit mesh-based surface representation because converting an implicit field to such a representation relies on the Marching Cubes algorithm, which cannot be differentiated with respect to the underlying implicit field. In this work, we remove this limitation and introduce a differentiable way to produce explicit surface mesh representations from Deep Implicit Fields. Our key insight is that by reasoning on how implicit field perturbations impact local surface geometry, one can ultimately differentiate the 3D location of surface samples with respect to the underlying deep implicit field. We exploit this to define DeepMesh - an end-to-end differentiable mesh representation that can vary its topology. We validate our theoretical insight through several applications: Single view 3D Reconstruction via Differentiable Rendering, Physically-Driven Shape Optimization, Full Scene 3D Reconstruction from Scans and End-to-End Training. In all cases our end-to-end differentiable parameterization gives us an edge over state-of-the-art algorithms.