PartEdit: Fine-Grained Image Editing using Pre-Trained Diffusion Models

TL;DR

PartEdit introduces a novel method for fine-grained, part-specific image editing using pre-trained diffusion models by learning special textual tokens and localization masks, significantly improving editing precision and user preference.

Contribution

This work expands pre-trained diffusion models to understand and edit specific object parts, enabling more precise and user-preferred image edits.

Findings

Outperforms existing editing methods on all metrics

Achieves 66-90% user preference in studies

Establishes a new benchmark and evaluation protocol for part editing

Abstract

We present the first text-based image editing approach for object parts based on pre-trained diffusion models. Diffusion-based image editing approaches capitalized on the deep understanding of diffusion models of image semantics to perform a variety of edits. However, existing diffusion models lack sufficient understanding of many object parts, hindering fine-grained edits requested by users. To address this, we propose to expand the knowledge of pre-trained diffusion models to allow them to understand various object parts, enabling them to perform fine-grained edits. We achieve this by learning special textual tokens that correspond to different object parts through an efficient token optimization process. These tokens are optimized to produce reliable localization masks at each inference step to localize the editing region. Leveraging these masks, we design feature-blending and adaptive thresholding strategies to execute the edits seamlessly. To evaluate our approach, we establish a benchmark and an evaluation protocol for part editing. Experiments show that our approach outperforms existing editing methods on all metrics and is preferred by users 66-90% of the time in conducted user studies.