DiffFashion: Reference-based Fashion Design with Structure-aware Transfer by Diffusion Models

TL;DR

DiffFashion introduces a diffusion model-based method for fashion design that transfers appearance onto clothing images while preserving structure, outperforming existing style transfer techniques in realism.

Contribution

The paper proposes a novel unsupervised, structure-aware diffusion approach utilizing semantic masks and ViT guidance for fashion design, addressing limitations of previous style transfer methods.

Findings

Outperforms state-of-the-art baselines in realism.

Effectively preserves clothing structure during transfer.

Generates more realistic fashion images.

Abstract

Image-based fashion design with AI techniques has attracted increasing attention in recent years. We focus on a new fashion design task, where we aim to transfer a reference appearance image onto a clothing image while preserving the structure of the clothing image. It is a challenging task since there are no reference images available for the newly designed output fashion images. Although diffusion-based image translation or neural style transfer (NST) has enabled flexible style transfer, it is often difficult to maintain the original structure of the image realistically during the reverse diffusion, especially when the referenced appearance image greatly differs from the common clothing appearance. To tackle this issue, we present a novel diffusion model-based unsupervised structure-aware transfer method to semantically generate new clothes from a given clothing image and a reference appearance image. In specific, we decouple the foreground clothing with automatically generated semantic masks by conditioned labels. And the mask is further used as guidance in the denoising process to preserve the structure information. Moreover, we use the pre-trained vision Transformer (ViT) for both appearance and structure guidance. Our experimental results show that the proposed method outperforms state-of-the-art baseline models, generating more realistic images in the fashion design task. Code and demo can be found at https://github.com/Rem105-210/DiffFashion.