3Deformer: A Common Framework for Image-Guided Mesh Deformation

TL;DR

3Deformer is a versatile, non-training framework for interactive 3D mesh editing guided by semantic images, effectively balancing shape accuracy, smoothness, and rigidity without dataset limitations.

Contribution

It introduces a training-free, image-guided 3D shape editing framework using differentiable rendering and hierarchical optimization, applicable to various objects.

Findings

Achieves state-of-the-art editing quality.

Effectively balances global and local shape features.

Demonstrates versatility across different object types.

Abstract

We propose 3Deformer, a general-purpose framework for interactive 3D shape editing. Given a source 3D mesh with semantic materials, and a user-specified semantic image, 3Deformer can accurately edit the source mesh following the shape guidance of the semantic image, while preserving the source topology as rigid as possible. Recent studies of 3D shape editing mostly focus on learning neural networks to predict 3D shapes, which requires high-cost 3D training datasets and is limited to handling objects involved in the datasets. Unlike these studies, our 3Deformer is a non-training and common framework, which only requires supervision of readily-available semantic images, and is compatible with editing various objects unlimited by datasets. In 3Deformer, the source mesh is deformed utilizing the differentiable renderer technique, according to the correspondences between semantic images and mesh materials. However, guiding complex 3D shapes with a simple 2D image incurs extra challenges, that is, the deform accuracy, surface smoothness, geometric rigidity, and global synchronization of the edited mesh should be guaranteed. To address these challenges, we propose a hierarchical optimization architecture to balance the global and local shape features, and propose further various strategies and losses to improve properties of accuracy, smoothness, rigidity, and so on. Extensive experiments show that our 3Deformer is able to produce impressive results and reaches the state-of-the-art level.