DPDEdit: Detail-Preserved Diffusion Models for Multimodal Fashion Image Editing

TL;DR

DPDEdit is a novel multimodal fashion image editing framework that accurately locates editing regions and preserves garment textures using a combination of grounded region prediction, texture injection, and refinement mechanisms.

Contribution

This work introduces DPDEdit, a new architecture that integrates multiple modalities and a texture refinement process for improved fashion image editing.

Findings

Outperforms existing methods in image fidelity

Achieves better coherence with multimodal inputs

Effectively preserves garment textures during editing

Abstract

Fashion image editing is a crucial tool for designers to convey their creative ideas by visualizing design concepts interactively. Current fashion image editing techniques, though advanced with multimodal prompts and powerful diffusion models, often struggle to accurately identify editing regions and preserve the desired garment texture detail. To address these challenges, we introduce a new multimodal fashion image editing architecture based on latent diffusion models, called Detail-Preserved Diffusion Models (DPDEdit). DPDEdit guides the fashion image generation of diffusion models by integrating text prompts, region masks, human pose images, and garment texture images. To precisely locate the editing region, we first introduce Grounded-SAM to predict the editing region based on the user's textual description, and then combine it with other conditions to perform local editing. To transfer the detail of the given garment texture into the target fashion image, we propose a texture injection and refinement mechanism. Specifically, this mechanism employs a decoupled cross-attention layer to integrate textual descriptions and texture images, and incorporates an auxiliary U-Net to preserve the high-frequency details of generated garment texture. Additionally, we extend the VITON-HD dataset using a multimodal large language model to generate paired samples with texture images and textual descriptions. Extensive experiments show that our DPDEdit outperforms state-of-the-art methods in terms of image fidelity and coherence with the given multimodal inputs.