NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editing

TL;DR

NeuMesh introduces a mesh-based neural implicit representation with disentangled geometry and texture codes, enabling advanced editing functionalities like mesh-guided modifications, texture swapping, filling, and painting, surpassing prior limited editing capabilities.

Contribution

The paper proposes a novel mesh-based neural implicit field with disentangled geometry and texture codes, facilitating fine-grained editing operations for 3D objects.

Findings

Superior representation quality demonstrated on real and synthetic data.

Effective editing functionalities including geometry and texture modifications.

Robust convergence achieved through distillation and fine-tuning mechanisms.

Abstract

Very recently neural implicit rendering techniques have been rapidly evolved and shown great advantages in novel view synthesis and 3D scene reconstruction. However, existing neural rendering methods for editing purposes offer limited functionality, e.g., rigid transformation, or not applicable for fine-grained editing for general objects from daily lives. In this paper, we present a novel mesh-based representation by encoding the neural implicit field with disentangled geometry and texture codes on mesh vertices, which facilitates a set of editing functionalities, including mesh-guided geometry editing, designated texture editing with texture swapping, filling and painting operations. To this end, we develop several techniques including learnable sign indicators to magnify spatial distinguishability of mesh-based representation, distillation and fine-tuning mechanism to make a steady convergence, and the spatial-aware optimization strategy to realize precise texture editing. Extensive experiments and editing examples on both real and synthetic data demonstrate the superiority of our method on representation quality and editing ability. Code is available on the project webpage: https://zju3dv.github.io/neumesh/.