Rethinking Garment Conditioning in Diffusion-based Virtual Try-On

TL;DR

This paper introduces Re-CatVTON, an efficient single UNet model for virtual try-on that achieves high fidelity with less computational cost by leveraging new conditioning strategies and theoretical insights.

Contribution

The paper proposes Re-CatVTON, a novel single UNet architecture with tailored guidance and ground-truth injection, outperforming previous models in efficiency and accuracy.

Findings

Re-CatVTON outperforms CatVTON in FID, KID, and LPIPS scores.

Re-CatVTON requires less computation and memory than Dual UNet models.

The model maintains comparable SSIM with improved overall performance.

Abstract

Virtual Try-On (VTON) is the task of synthesizing an image of a person wearing a target garment, conditioned on a person image and a garment image. While diffusion-based VTON models featuring a Dual UNet architecture demonstrate superior fidelity compared to single UNet models, they incur substantial computational and memory overhead due to their heavy structure. In this study, through visualization analysis and theoretical analysis, we derived three hypotheses regarding the learning of context features to condition the denoising process. Based on these hypotheses, we developed Re-CatVTON, an efficient single UNet model that achieves high performance. We further enhance the model by introducing a modified classifier-free guidance strategy tailored for VTON's spatial concatenation conditioning, and by directly injecting the ground-truth garment latent derived from the clean garment latent to prevent the accumulation of prediction error. The proposed Re-CatVTON significantly improves performance compared to its predecessor (CatVTON) and requires less computation and memory than the high-performance Dual UNet model, Leffa. Our results demonstrate improved FID, KID, and LPIPS scores, with only a marginal decrease in SSIM, establishing a new efficiency-performance trade-off for single UNet VTON models.