As-Plausible-As-Possible: Plausibility-Aware Mesh Deformation Using 2D Diffusion Priors

TL;DR

This paper introduces APAP, a mesh deformation method that uses 2D diffusion priors and differentiable rendering to produce plausible deformations aligned with user edits, outperforming previous geometry-focused techniques.

Contribution

The paper proposes a novel plausibility-aware mesh deformation framework leveraging 2D diffusion priors and differentiable rendering, with fine-tuning for identity preservation.

Findings

Qualitative and quantitative improvements over prior methods.

Effective use of 2D diffusion priors for mesh plausibility.

Enhanced preservation of mesh identity during deformation.

Abstract

We present As-Plausible-as-Possible (APAP) mesh deformation technique that leverages 2D diffusion priors to preserve the plausibility of a mesh under user-controlled deformation. Our framework uses per-face Jacobians to represent mesh deformations, where mesh vertex coordinates are computed via a differentiable Poisson Solve. The deformed mesh is rendered, and the resulting 2D image is used in the Score Distillation Sampling (SDS) process, which enables extracting meaningful plausibility priors from a pretrained 2D diffusion model. To better preserve the identity of the edited mesh, we fine-tune our 2D diffusion model with LoRA. Gradients extracted by SDS and a user-prescribed handle displacement are then backpropagated to the per-face Jacobians, and we use iterative gradient descent to compute the final deformation that balances between the user edit and the output plausibility. We evaluate our method with 2D and 3D meshes and demonstrate qualitative and quantitative improvements when using plausibility priors over geometry-preservation or distortion-minimization priors used by previous techniques. Our project page is at: https://as-plausible-aspossible.github.io/