Texture-Preserving Diffusion Models for High-Fidelity Virtual Try-On

TL;DR

This paper introduces a texture-preserving diffusion model for virtual try-on that improves fidelity without extra encoders by concatenating images and predicting inpainting masks, outperforming existing methods.

Contribution

The paper presents a novel diffusion-based virtual try-on approach that enhances texture transfer efficiency and accuracy without additional encoders, using image concatenation and mask prediction.

Findings

Outperforms state-of-the-art methods on VITON and VITON-HD datasets.

Enables efficient and accurate texture transfer with a single compact model.

Supports various try-on tasks like garment-to-person and person-to-person.

Abstract

Image-based virtual try-on is an increasingly important task for online shopping. It aims to synthesize images of a specific person wearing a specified garment. Diffusion model-based approaches have recently become popular, as they are excellent at image synthesis tasks. However, these approaches usually employ additional image encoders and rely on the cross-attention mechanism for texture transfer from the garment to the person image, which affects the try-on's efficiency and fidelity. To address these issues, we propose an Texture-Preserving Diffusion (TPD) model for virtual try-on, which enhances the fidelity of the results and introduces no additional image encoders. Accordingly, we make contributions from two aspects. First, we propose to concatenate the masked person and reference garment images along the spatial dimension and utilize the resulting image as the input for the diffusion model's denoising UNet. This enables the original self-attention layers contained in the diffusion model to achieve efficient and accurate texture transfer. Second, we propose a novel diffusion-based method that predicts a precise inpainting mask based on the person and reference garment images, further enhancing the reliability of the try-on results. In addition, we integrate mask prediction and image synthesis into a single compact model. The experimental results show that our approach can be applied to various try-on tasks, e.g., garment-to-person and person-to-person try-ons, and significantly outperforms state-of-the-art methods on popular VITON, VITON-HD databases.