Drag within Prior Distribution: Text-Conditioned Point-Based Image Editing within Distribution Constraints

TL;DR

This paper introduces a novel diffusion-based image editing method that maintains distribution consistency and semantic alignment during localized manipulations, addressing issues of artifacts and ambiguity in prior approaches.

Contribution

It proposes a CLIP-guided intermediate step evaluation, a prior-preservation loss for distribution consistency, and a directionally-weighted point tracking mechanism for improved accuracy and efficiency.

Findings

Enhanced semantic alignment in edited images.

Reduced artifacts and deviations from original data distribution.

Improved editing accuracy and speed in fine-grained tasks.

Abstract

Diffusion-based point editing methods have gained significant traction in image editing tasks due to their ability to manipulate image semantics and fine details by applying localized perturbations on the manifold of noise latent. However, these approaches face several limitations. Traditional point-based editing relies on pairs of handle and target points to define motion trajectories, which can introduce ambiguity or unnecessary alterations. Furthermore, when the distance between the handle and target points is large, the accumulated perturbations often cause the noise latent deviation from inversion score trajectory, resulting in unnatural artifacts. To address these issues in global editing tasks, we introduce a CLIP-based model to evaluate and guide intermediate editing steps, ensuring that the generated results remain both semantically aligned. Additionally, we propose a prior-preservation loss that constrains the optimized latent code to stay within the sampling space of the diffusion prior, improving consistency with the original data distribution, to ensure the model generates images along a familiar score trajectory. For fine-grained tasks, we present a directionally-weighted point tracking mechanism that steers the editing process toward the target direction within similar feature regions. This improves both the tracking accuracy and generation quality, while also reducing the editing time.