Leveraging Off-the-shelf Diffusion Model for Multi-attribute Fashion Image Manipulation

TL;DR

This paper introduces a versatile, diffusion model-based framework for multi-attribute fashion image editing, overcoming limitations of prior GAN-based methods by enabling scalable, generic attribute manipulation without requiring separate models for each attribute or dataset.

Contribution

The authors propose a novel diffusion model-based approach for multi-attribute fashion image editing that is scalable, generic, and does not rely on separate models for each attribute or dataset.

Findings

Effective multi-attribute editing with convincing quality.

Outperforms conventional methods in challenging settings.

Utilizes attention-pooling for efficient classifier guidance.

Abstract

Fashion attribute editing is a task that aims to convert the semantic attributes of a given fashion image while preserving the irrelevant regions. Previous works typically employ conditional GANs where the generator explicitly learns the target attributes and directly execute the conversion. These approaches, however, are neither scalable nor generic as they operate only with few limited attributes and a separate generator is required for each dataset or attribute set. Inspired by the recent advancement of diffusion models, we explore the classifier-guided diffusion that leverages the off-the-shelf diffusion model pretrained on general visual semantics such as Imagenet. In order to achieve a generic editing pipeline, we pose this as multi-attribute image manipulation task, where the attribute ranges from item category, fabric, pattern to collar and neckline. We empirically show that conventional methods fail in our challenging setting, and study efficient adaptation scheme that involves recently introduced attention-pooling technique to obtain a multi-attribute classifier guidance. Based on this, we present a mask-free fashion attribute editing framework that leverages the classifier logits and the cross-attention map for manipulation. We empirically demonstrate that our framework achieves convincing sample quality and attribute alignments.