Preserving Identity with Variational Score for General-purpose 3D Editing

TL;DR

Piva is a new optimization method that improves 3D and image editing by preserving identity and details, overcoming limitations of previous diffusion-based techniques.

Contribution

It introduces a score distillation term that maintains identity during editing, enabling stable, versatile, and high-quality 3D and 2D editing without pre-training or masking.

Findings

Effective zero-shot editing of images and neural fields.

Preserves identity and details better than previous methods.

Achieves competitive results on standard benchmarks.

Abstract

We present Piva (Preserving Identity with Variational Score Distillation), a novel optimization-based method for editing images and 3D models based on diffusion models. Specifically, our approach is inspired by the recently proposed method for 2D image editing - Delta Denoising Score (DDS). We pinpoint the limitations in DDS for 2D and 3D editing, which causes detail loss and over-saturation. To address this, we propose an additional score distillation term that enforces identity preservation. This results in a more stable editing process, gradually optimizing NeRF models to match target prompts while retaining crucial input characteristics. We demonstrate the effectiveness of our approach in zero-shot image and neural field editing. Our method successfully alters visual attributes, adds both subtle and substantial structural elements, translates shapes, and achieves competitive results on standard 2D and 3D editing benchmarks. Additionally, our method imposes no constraints like masking or pre-training, making it compatible with a wide range of pre-trained diffusion models. This allows for versatile editing without needing neural field-to-mesh conversion, offering a more user-friendly experience.