WildVidFit: Video Virtual Try-On in the Wild via Image-Based Controlled Diffusion Models

TL;DR

WildVidFit introduces a novel image-based controlled diffusion model for video virtual try-on, generating realistic, temporally coherent videos conditioned on garment descriptions and human motion, overcoming limitations of traditional methods.

Contribution

It presents a one-stage diffusion-based approach trained on still images that maintains temporal coherence in video try-on, reducing data and computational requirements.

Findings

Effective in generating fluid, coherent videos on multiple datasets.

Outperforms traditional warping and blending methods.

Leverages pre-trained models for improved temporal consistency.

Abstract

Video virtual try-on aims to generate realistic sequences that maintain garment identity and adapt to a person's pose and body shape in source videos. Traditional image-based methods, relying on warping and blending, struggle with complex human movements and occlusions, limiting their effectiveness in video try-on applications. Moreover, video-based models require extensive, high-quality data and substantial computational resources. To tackle these issues, we reconceptualize video try-on as a process of generating videos conditioned on garment descriptions and human motion. Our solution, WildVidFit, employs image-based controlled diffusion models for a streamlined, one-stage approach. This model, conditioned on specific garments and individuals, is trained on still images rather than videos. It leverages diffusion guidance from pre-trained models including a video masked autoencoder for segment smoothness improvement and a self-supervised model for feature alignment of adjacent frame in the latent space. This integration markedly boosts the model's ability to maintain temporal coherence, enabling more effective video try-on within an image-based framework. Our experiments on the VITON-HD and DressCode datasets, along with tests on the VVT and TikTok datasets, demonstrate WildVidFit's capability to generate fluid and coherent videos. The project page website is at wildvidfit-project.github.io.