MADiff: Text-Guided Fashion Image Editing with Mask Prediction and Attention-Enhanced Diffusion

TL;DR

MADiff introduces a novel approach for fashion image editing that combines mask prediction and attention-enhanced diffusion to improve localization and editing strength, addressing limitations of existing text-guided models in the fashion domain.

Contribution

The paper proposes MADiff, a new model with MaskNet for accurate region localization and an Attention-Enhanced Diffusion Model for stronger editing, tailored for fashion image editing.

Findings

Accurately predicts editing regions in fashion images.

Significantly improves editing magnitude over state-of-the-art methods.

Constructed the Fashion-E dataset for benchmarking fashion image editing.

Abstract

Text-guided image editing model has achieved great success in general domain. However, directly applying these models to the fashion domain may encounter two issues: (1) Inaccurate localization of editing region; (2) Weak editing magnitude. To address these issues, the MADiff model is proposed. Specifically, to more accurately identify editing region, the MaskNet is proposed, in which the foreground region, densepose and mask prompts from large language model are fed into a lightweight UNet to predict the mask for editing region. To strengthen the editing magnitude, the Attention-Enhanced Diffusion Model is proposed, where the noise map, attention map, and the mask from MaskNet are fed into the proposed Attention Processor to produce a refined noise map. By integrating the refined noise map into the diffusion model, the edited image can better align with the target prompt. Given the absence of benchmarks in fashion image editing, we constructed a dataset named Fashion-E, comprising 28390 image-text pairs in the training set, and 2639 image-text pairs for four types of fashion tasks in the evaluation set. Extensive experiments on Fashion-E demonstrate that our proposed method can accurately predict the mask of editing region and significantly enhance editing magnitude in fashion image editing compared to the state-of-the-art methods.