ShapeUP: Scalable Image-Conditioned 3D Editing

TL;DR

ShapeUP introduces a scalable 3D editing framework that uses a supervised latent translation approach with a 3D Diffusion Transformer, enabling precise, controllable, and consistent 3D asset modifications.

Contribution

It presents a novel image-conditioned 3D editing method that leverages a pretrained 3D foundation model and supervised training for improved control and scalability.

Findings

Outperforms existing methods in identity preservation and edit fidelity.

Enables fine-grained, mask-free local and global 3D edits.

Maintains structural consistency with original assets.

Abstract

Recent advancements in 3D foundation models have enabled the generation of high-fidelity assets, yet precise 3D manipulation remains a significant challenge. Existing 3D editing frameworks often face a difficult trade-off between visual controllability, geometric consistency, and scalability. Specifically, optimization-based methods are prohibitively slow, multi-view 2D propagation techniques suffer from visual drift, and training-free latent manipulation methods are inherently bound by frozen priors and cannot directly benefit from scaling. In this work, we present ShapeUP, a scalable, image-conditioned 3D editing framework that formulates editing as a supervised latent-to-latent translation within a native 3D representation. This formulation allows ShapeUP to build on a pretrained 3D foundation model, leveraging its strong generative prior while adapting it to editing through supervised training. In practice, ShapeUP is trained on triplets consisting of a source 3D shape, an edited 2D image, and the corresponding edited 3D shape, and learns a direct mapping using a 3D Diffusion Transformer (DiT). This image-as-prompt approach enables fine-grained visual control over both local and global edits and achieves implicit, mask-free localization, while maintaining strict structural consistency with the original asset. Our extensive evaluations demonstrate that ShapeUP consistently outperforms current trained and training-free baselines in both identity preservation and edit fidelity, offering a robust and scalable paradigm for native 3D content creation.