Posterior Distillation Sampling

TL;DR

Posterior Distillation Sampling (PDS) is a new optimization technique that enables effective parametric image editing by balancing attribute modification and content preservation using diffusion models.

Contribution

PDS reformulates 2D image editing methods into an optimization framework that aligns source and target stochastic latents in diffusion models, extending editing capabilities to parameter spaces.

Findings

Effective in Neural Radiance Fields editing

Works across diverse parameter spaces

Maintains source identity during editing

Abstract

We introduce Posterior Distillation Sampling (PDS), a novel optimization method for parametric image editing based on diffusion models. Existing optimization-based methods, which leverage the powerful 2D prior of diffusion models to handle various parametric images, have mainly focused on generation. Unlike generation, editing requires a balance between conforming to the target attribute and preserving the identity of the source content. Recent 2D image editing methods have achieved this balance by leveraging the stochastic latent encoded in the generative process of diffusion models. To extend the editing capabilities of diffusion models shown in pixel space to parameter space, we reformulate the 2D image editing method into an optimization form named PDS. PDS matches the stochastic latents of the source and the target, enabling the sampling of targets in diverse parameter spaces that align with a desired attribute while maintaining the source's identity. We demonstrate that this optimization resembles running a generative process with the target attribute, but aligning this process with the trajectory of the source's generative process. Extensive editing results in Neural Radiance Fields and Scalable Vector Graphics representations demonstrate that PDS is capable of sampling targets to fulfill the aforementioned balance across various parameter spaces.