MeshUp: Multi-Target Mesh Deformation via Blended Score Distillation

TL;DR

MeshUp introduces a novel method for deforming 3D meshes towards multiple target concepts, allowing intuitive control over local regions and combining text or image-based concepts through a new score distillation technique.

Contribution

The paper presents Blended Score Distillation, a new approach enabling multi-target mesh deformation with localized control using diffusion models and probabilistic ROI maps.

Findings

Effectively deforms meshes towards multiple concepts.

Allows local and global control of mesh deformation.

Demonstrates versatility across various mesh types.

Abstract

We propose MeshUp, a technique that deforms a 3D mesh towards multiple target concepts, and intuitively controls the region where each concept is expressed. Conveniently, the concepts can be defined as either text queries, e.g., "a dog" and "a turtle," or inspirational images, and the local regions can be selected as any number of vertices on the mesh. We can effectively control the influence of the concepts and mix them together using a novel score distillation approach, referred to as the Blended Score Distillation (BSD). BSD operates on each attention layer of the denoising U-Net of a diffusion model as it extracts and injects the per-objective activations into a unified denoising pipeline from which the deformation gradients are calculated. To localize the expression of these activations, we create a probabilistic Region of Interest (ROI) map on the surface of the mesh, and turn it into 3D-consistent masks that we use to control the expression of these activations. We demonstrate the effectiveness of BSD empirically and show that it can deform various meshes towards multiple objectives. Our project page is at https://threedle.github.io/MeshUp.