VITON-DiT: Learning In-the-Wild Video Try-On from Human Dance Videos via Diffusion Transformers

TL;DR

VITON-DiT introduces a diffusion transformer-based framework for realistic in-the-wild video try-on, capable of handling casual videos with complex poses without requiring paired training data.

Contribution

First DiT-based video try-on framework that works with unpaired dance videos, using novel training strategies and a new benchmark dataset.

Findings

Outperforms existing methods in spatio-temporal consistency

Effective on casual videos with complex human poses

Does not require paired training datasets

Abstract

Video try-on stands as a promising area for its tremendous real-world potential. Prior works are limited to transferring product clothing images onto person videos with simple poses and backgrounds, while underperforming on casually captured videos. Recently, Sora revealed the scalability of Diffusion Transformer (DiT) in generating lifelike videos featuring real-world scenarios. Inspired by this, we explore and propose the first DiT-based video try-on framework for practical in-the-wild applications, named VITON-DiT. Specifically, VITON-DiT consists of a garment extractor, a Spatial-Temporal denoising DiT, and an identity preservation ControlNet. To faithfully recover the clothing details, the extracted garment features are fused with the self-attention outputs of the denoising DiT and the ControlNet. We also introduce novel random selection strategies during training and an Interpolated Auto-Regressive (IAR) technique at inference to facilitate long video generation. Unlike existing attempts that require the laborious and restrictive construction of a paired training dataset, severely limiting their scalability, VITON-DiT alleviates this by relying solely on unpaired human dance videos and a carefully designed multi-stage training strategy. Furthermore, we curate a challenging benchmark dataset to evaluate the performance of casual video try-on. Extensive experiments demonstrate the superiority of VITON-DiT in generating spatio-temporal consistent try-on results for in-the-wild videos with complicated human poses.