Dress&Dance: Dress up and Dance as You Like It - Technical Preview

TL;DR

Dress&Dance is a novel video diffusion framework that creates high-quality, realistic virtual try-on videos of users wearing various garments, with synchronized movement based on reference videos, using a single user image and multi-modal conditioning.

Contribution

The paper introduces CondNet, a new attention-based conditioning network that unifies multi-modal inputs for improved garment registration and motion fidelity in virtual try-on videos.

Findings

Outperforms existing open source and commercial solutions.

Generates high-quality 5-second videos at 24 FPS and 1152x720 resolution.

Supports diverse garments and simultaneous try-on in a single pass.

Abstract

We present Dress&Dance, a video diffusion framework that generates high quality 5-second-long 24 FPS virtual try-on videos at 1152x720 resolution of a user wearing desired garments while moving in accordance with a given reference video. Our approach requires a single user image and supports a range of tops, bottoms, and one-piece garments, as well as simultaneous tops and bottoms try-on in a single pass. Key to our framework is CondNet, a novel conditioning network that leverages attention to unify multi-modal inputs (text, images, and videos), thereby enhancing garment registration and motion fidelity. CondNet is trained on heterogeneous training data, combining limited video data and a larger, more readily available image dataset, in a multistage progressive manner. Dress&Dance outperforms existing open source and commercial solutions and enables a high quality and flexible try-on experience.

Tables

Table 1. Table 1 : In the quantitative evaluations, Dress&Dance significantly outperforms the open-sourced baselines under most of the metrics, while achieving comparable or even better quality metrics than commercial models Kling AI [ 1 ] and Ray2 [ 19 ] . The bold numbers mark the best for each metric and the underlined numbers mark the second best.

Method Property		Method		Evaluation with Ground Truth on Captured Dataset
Ours	Commercial			PSNR$\uparrow$	SSIM$\uparrow$	${\text{LPIPS}}_{\text{VGG}}\downarrow$	${\text{LPIPS}}_{\text{AlexNet}}\downarrow$
		TPD	+ CogVideoX I2V	14.47	0.8305	0.2840	0.2461
		OOTDiffusion	+ CogVideoX I2V	14.68	0.8282	0.2779	0.2457
		ML-VTON	+ CogVideoX I2V	14.49	0.8270	0.2961	0.2520
✓		Dress&Dance Image Try-On	+ CogVideoX I2V	17.26	0.8515	0.2812	0.1635
	✓	Kling Image Try-On	+ Kling Video 1.6	17.33	0.8651	0.2296	0.1683
✓	✓	Dress&Dance Image Try-On	+ Kling Video 1.6	15.21	0.8297	0.2835	0.2066
✓	✓	Dress&Dance Image Try-On	+ Ray2	15.16	0.8290	0.2938	0.1990
✓		Dress&Dance, Direct Train		17.14	0.8678	0.2854	0.1338
✓		Dress&Dance		22.41	0.9038	0.2382	0.0624

Table 2. Table 2 : Our Dress&Dance significantly outperforms all the baselines in garment fidelity in virtual try-on ( GPT Try-On \text{GPT}_{\text{Try-On}} ), showing superior try-on capability, while achieving highly comparable or even better visual quality in other metrics to powerful commercial baselines Kling Video 1.6 [ 1 ] and Ray2 [ 19 ] , which outperforms all open-sourced baselines with large margin.

Method Property		Method		Evaluation of Try-On and Visual Quality
Ours	Commercial			${\text{GPT}}_{\text{Try-On}}\uparrow$	${\text{GPT}}_{\text{User}}\uparrow$	${\text{GPT}}_{\text{Motion}}\uparrow$	${\text{GPT}}_{\text{Visual}}\uparrow$	${\text{GPT}}_{\text{Overall}}\uparrow$	${\text{FID}}_{\text{Internet}}\downarrow$	${\text{FID}}_{\text{Captured}}\downarrow$
		TPD	+ CogVideoX I2V	69.67	73.98	68.15	65.45	68.64	1146	753
		OOTDiffusion	+ CogVideoX I2V	70.57	76.41	68.71	70.78	70.76	1089	884
		ML-VTON	+ CogVideoX I2V	69.69	76.62	68.50	69.95	70.50	1185	739
✓		Dress&Dance Image Try-On	+ CogVideoX I2V	85.05	87.54	75.68	76.88	80.71	1109	760
	✓	Kling Image Try-On	+ Kling Video 1.6	80.10	89.97	85.48	84.70	84.38	982	655
✓	✓	Dress&Dance Image Try-On	+ Kling Video 1.6	86.85	89.77	82.59	84.94	85.85	1008	700
✓	✓	Dress&Dance Image Try-On	+ Ray2	86.79	88.99	79.31	83.48	84.18	1094	735
✓		Dress&Dance, Direct Train		79.48	78.47	72.00	71.24	74.85	1073	788
✓		Dress&Dance		87.41	88.89	80.35	84.48	84.95	1055	691

Full text

1 Introduction

Imagine yourself trying a new dress in a fitting room. It is natural to move around to get a feel for how the garment fits and flows. Current computational methods that enable virtual garment try-on typically generate static 2D images. Viewing oneself in a single image does not provide the full and expressive experience of a try-on. In this paper, we introduce Dress&Dance, a video diffusion framework that generates a $5$-second, $24$ FPS video at $1152\times 720$ resolution of a user wearing a desired garment. The motion in the generated video is guided by an example video selected by the user. Our approach allows the user to choose the garment (** Dress**) and control how they would like to animate (** Dance**) with it, as illustrated in Fig. 1. Dress&Dance supports a wide range of tops, bottoms, and one-piece garments – as well as their combinations at once. Our model allows for try-ons of clothing borrowed from another human and garments and accessories modification using text prompts. Fig. 2 highlights a few applications of Dress&Dance.

The emergence of diffusion models [20] has allowed the generation of videos from text [9, 30] and images [2, 32, 1]. Generating videos from text requires highly detailed descriptions; otherwise, the model struggles to capture essential details. In our context, it is difficult to describe the content textually such that both the user’s appearance and the integrity of the target garment are preserved in the generated results. To sidestep the problem, one strategy is to first perform a single image try-on [28, 26, 24, 15] and then animate the output with video diffusion models like CogVideoX [30]. We observe temporally incoherent outputs due to error propagation from the first frame (Fig. 3-(a)), particularly when there are significant pose changes, leading to occlusions of the garment or human body. Another challenge is to instruct the machine learning model to generate desired motion. Simple movements, such as turn-around, can be described textually. Capturing more nuanced movements is difficult with text, even when using state-of-the-art commercial methods such as Kling [1] and Ray2 [19], as shown in Fig. 3-(b). We overcome these challenges by learning an end-to-end approach that leverages exemplar videos to describe nuanced motion. Our results faithfully preserve garment and user appearance details and strictly follow the direction, as shown in Fig. 3-(c).

The key to our work is a conditioning network, CondNet, that leverages cross-attention for conditioning on multiple modalities (text, images, and videos). By converting heterogeneous conditions into homogeneous attention sequences, we can concatenate them with the sequences of video and text for cross-attention, enabling the model to handle them in a unified manner. CondNet implicitly connects all pairs of the input image’s pixels to the generated video’s pixels through cross-attention. This allows us to better transfer garment details, supports multiple garment types, including garment combination simultaneously, and is robust to varying garment capture methods. Whether the garment is casually captured as a flat image or worn by an individual, our model can effectively handle both scenarios.

Due to the high compute cost of attention modules in generating high-resolution visual output, we design a stage-wise, multi-phase, resolution progressive training strategy to effectively and efficiently train the model. First, we adopt a curriculum garment warm-up learning to guide the coarse garment try-on location on the user body and a progressive resolution training to preserve the user and garment identities. Secondly, we further boost the resolution and quality, by introducing an auto-regressive video refiner stage to upsample the video from $8$ FPS to $24$ FPS while further refining the appearance details. We show ablation results that demonstrate the effectiveness of this training strategy.

Our contributions: (1) We propose CondNets, a simple yet effective conditioning strategy that unifies multi-modal inputs through cross-attention, improving both garment registration and overall try-on quality. (2) We introduce compute-effective and data-efficient training strategies that allow us to generate high-resolution outputs with limited compute and data. (3) We show experimentally that our simple method can effectively handle arbitrary garments and is robust to various garment capture methods, enabling the user to simultaneously try on a set of garments or point to a garment worn by another person and perform a try-on. Our method achieves state-of-the-art quality in various challenging scenarios, outperforming open source baselines and commercial models such as Kling [1] and Ray2 [19].

2 Related Work

Videos from a Single Image. As a representative method, Stable Video Diffusion (SVD) [2] generates short videos from a single image. However, it supports only landscape videos and is limited to short video lengths. Similarly, I2VGen-XL [32] uses image and text input to generate short videos, but is restricted to landscape formats. A more flexible solution, CogVideoX-I2V [30], supports portrait videos and allows image-to-video synthesis with text guidance, providing more control over generated content. Commercial models, such as Kling Video 1.6 [1] and Ray2 [19], offer a similar capability for image-to-video synthesis, supporting portrait videos with image and text inputs, enabling a more customizable output. These methods focus on generating video sequences from still images but do not address garment try-on specifically or the need for motion-guided generation. Note that Kling has a standalone image try-on model which we use for analysis in this work.

Single 2D Image Try-On. This is the most popular virtual try-on work stream and supports both one or multiple target garment(s) to produce a single try-on image as output [22, 13, 28, 26, 24, 15, 29, 35, 7, 3, 34, 25]. These methods typically require paired training data consisting of the garment and corresponding human model images wearing that garment. To prevent information leakage from such paired training data, they extract garment agnostics human intermediates, such as the pose or try-on region mask, from the human model images, and train the model to resynthesize the original RGB human model images in a self-supervised setting. However, at inference time, the model is given the unpaired setting where it needs to try-on a chosen garment on a user image wearing a different garment, making inference harder than training due to information leakage at training stage. Moreover, the use of the intermediate representations also propagate their pre-processing artifacts to the try-on results. Our method sidesteps both problems using synthetically generated unpaired triplets to not only eliminate the need for these intermediates, but also to enable a consistent training setting as evaluation, potentially improving the try-on results in the evaluation.

Video-to-Video Translation. Video-to-video translation methods have explored both supervised and unsupervised approaches to generate temporally consistent videos. VideoShop [5] is a training-free approach that relies on a pre-trained diffusion model to edit the first frame of a video, which is then propagated across subsequent frames to maintain temporal consistency. Similarly, BIVDiff [21] performs per-frame editing, refining the video using a pre-trained diffusion model to ensure consistency across frames. Another relevant approach is CogVideoX-V2V [30], employing a pre-trained text-to-video diffusion model combined with SDEdit [17] for text-guided video editing. These methods focus on video editing and manipulation, but do not address the specific needs of garment try-on or the combination of motion and garment generation. Dress&Dance seamlessly integrates garment try-on within video generation, allowing users to choose a desired garment and manipulate motion, effectively combining the adaptability of video-to-video translation with the added functionality of virtual try-on.

Video Virtual Try-On (VVT). Early work of VVT [4, 33, 11, 14] utilizes generative adversarial networks (GANs) [8] to perform virtual try-on, and warp previous frames to new frames to form videos. After the emergence of diffusion models [2], follow-up work [27, 23, 6, 10, 12] leverages image or video diffusion models for VVT. However, these methods produce low-resolution (lower than 512$\times$512) VVT videos with fewer frames. Additionally, the diffusion-based methods work only on UNet-based diffusion models, which limits their scalability. In comparison, our Dress&Dance generates high 1152$\times$720 resolution, $121$-frame videos, through a powerful DiT-based video diffusion model.

3 Dress&Dance: Methodology Overview

We propose Dress&Dance, a novel video diffusion framework for virtual try-on, as illustrated in Fig. 5. Given a user image, a garment or garment set to try on, a reference motion video, and an optional text prompt, Dress&Dance synthesizes realistic videos of the user performing the indicated motion while wearing the desired garments. As shown in Fig. 5-(a), all inputs are encoded into token sequences and fed into a unified diffusion backbone. To support rich multi-modal conditioning, we introduce CondNets (Fig. 5-(b)), modular attention-based adapters that inject garment, user, and motion cues into the diffusion process. A dedicated video refiner (Fig. 5-(c)), fine-tuned from our main Dress&Dance model, further upsamples the initial 8-FPS output to 24 FPS, while enhancing visual quality and removing artifacts.

To enable supervised training despite limited paired video data, we construct synthetic triplets, which eliminates the need for intermediates like “agnostic masks” or “Dense Poses” used in existing works, bridging the gap between training and inference formats. During optimization, we employ two key strategies: (1) a garment warm-up training phase based on curriculum learning, which guides the model to quickly learn garment registration—a core capability for try-on; and (2) a multi-stage progressive training scheme that gradually increases resolution and condition complexity, improving training stability and convergence. Together, these designs allow Dress&Dance to deliver high-fidelity and controllable try-on results under diverse settings.

4 Experiments

We study the performance of Dress&Dance using three different try-on modes (Fig. 2): (1) single garment mode allows the user to try on the garment shown in a single flat garment image, (2) “a set of garments” or multiple garment mode allows simultaneous try-on of all the garments provided as flat garment images, and (3) garment transfer mode allows to transfer a garment from an existing image through segmentation.

Datasets. We curated two video datasets for training and evaluation. (1) Internet video dataset is constructed by crawling publicly available fashion videos with paired flat garment images, which contains around $80K$ garment-video pairs. (2) Our captured video dataset is constructed by hiring 183 human models to record the try-on videos for various sets of garments. Each model records the try-on videos for around 100 different sets of garments, so we can construct unpaired multi-garment try-on data with ground truth, by cross-matching the frames of different garments of the same person. We also collected an image dataset on the Internet with around 4M pairs of garment images paired for hybrid training. We divided each dataset into training and test subsets and performed training on the combination of all training subsets. For evaluation, we either sample garments and models from the evaluation subsets or use other garments, human images, and motion reference videos from the Internet.

Baselines. As there are no publicly available methods that support virtual try-on video generation, we compose the baseline methods using an “image try-on + image animation” pipeline: we first apply virtual try-on to the user image with a state-of-the-art image try-on method and then animate it according to the prompt with a video generation method. We consider two different types of baselines: (1) open-source models. To perform single image try-on, we use state-of-the-art image try-on methods, TPD [28] and OOTDiffusion [26]. We also use “ML-VTON”, an engineered version of GP-VTON [24] that combines the original warping module with the HR-VTON [15] try-on module. For comparison, we also provide the single image try-on generated by Dress&Dance as a single-frame video. We then animate these try-on images with CogVideoX 1.5 I2V [30], one of the best open-source image-to-video (I2V) generation models available with text prompt descriptions. We cannot use other methods, including Stable Video Diffusion [2] and I2VGen-XL [32], as they do not support portrait videos well. (2) commercial models. We also compare with two state-of-the-art commercial methods for video generation: Kling Video 1.6 [1] and Ray2 [19]. We once again perform image try-on with our Dress&Dance on the user image and use these models to animate according to the text prompt. Kling AI’s platform also provides an image try-on model. We also use it to perform a try-on for the subsequent video generation. (3) video try-on baselines. We compare with ViViD [6], WildFit [10], Tunnel Try-On [27], GPD-VVTO [23], and ClothFormer [11] in “single-garment” setting, and Fashion-VDM [12] in the aligned “garment transfer” try-on task. Due to the lack of publicly available code, we compare our Dress&Dance with the tasks presented on their website.

4.1 Results and Analysis

Qualitative Comparison. We show the qualitative results in Figs. 3, 4, and 7. Fig. 3 show the try-on results in the single garment mode. More results are on our project page. In the user image of Fig. 3, the hand of the user covers the top-right corner of the garment. Therefore, any baseline, including Kling [1] and Ray2 [19], that first apply try-on to the user image and then animate it with another model, failed to generate the correct top-right garment pattern. The information contained in the occluded garment is unable to be recovered, as the animation model does not consider the garment image. On the contrary, Dress&Dance sees the garment image during the generation of the entire video and generates correct and consistent video try-on results. On our project page, we evaluate our model with challenging dancing motions with two different dances, which are difficult to describe in text. This leads to the failure of all the baselines. Using the skeleton video extracted from the dancing video, our Dress&Dance is able to generate an accurate and smooth dancing motion with the correct try-on results.

The results of the multiple garment mode are shown in Fig. 7. Dress&Dance is able to perform the virtual try-on of both top and bottom garments simultaneously, without any explicit information indicating the garment’s type (e.g., which one is the top or the bottom). Regardless of the types and the orders of garments to try on, Dress&Dance consistently generates high-quality try-on results. Kling AI, though officially supports multi-garment try-on, incorrectly tries on the trousers as a dress.

In Fig. 8, we compare our model with multiple video try-on baselines in single garment mode; in Fig. 4, we compare our model with Fashion-VDM [12] in garment transfer mode. Dress&Dance generates the videos at a higher resolution ($1152\times 720$), which better preserves the details and textures of the garments, and superior quality in rendering semi-transparent garments, while any other baseline’s results are limited to a lower $512\times 384$ resolution with blurred and gloomy textures.

Quantitative Comparison. We show the quantitative comparison in Tab. 1, where we conduct experiments on tasks constructed from our captured dataset, and compare the generated video with the ground truth using PSNR, SSIM, and LPIPS [31]. Dress&Dance achieves results better than most methods. It also remains competitive to commercial models, Kling Video 1.6 [1] and Ray2 [19]. For our Internet dataset, it is challenging to evaluate the try-on quality and visual quality, due to so many degrees of freedom in the generation. Inspired by VQAScore [16], we leverage GPT [18]’s strong capability in vision-language reasoning to grade the generated videos from the following aspects: garment try-on fidelity and quality ($\text{GPT}_{\text{Try-On}}$), user appearance fidelity ($\text{GPT}_{\text{User}}$), human and garment motion quality ($\text{GPT}_{\text{Motion}}$), visual quality ($\text{GPT}_{\text{Visual}}$), and finally the overall quality considering all the aspects above ($\text{GPT}_{\text{Overall}}$). We provide detailed instructions and rubrics to guide the GPT to grade, while re-grading each video $40$ times and taking the average to avoid randomness and ensure fairness. The results are shown in Tab. 2.

Virtual Try-On Capability. As shown in the “$\text{GPT}_{\text{Try-On}}$” row, our Dress&Dance significantly outperforms all the baseline models in garment fidelity and try-on quality. Also, using our Dress&Dance for image try-on in baseline’s pipeline consistently improves the try-on quality with large margins, compared to the one using other image try-on methods, including Kling AI [1]. This shows that our Dress&Dance has superior capability in virtual try-on.

Visual Quality. In all other metrics, our Dress&Dance achieves highly comparable results as the commercial models. Notably, as the commercial models are trained with much more video data and have the flexibility to generate motions only constrained by the text prompt, they are intrinsically easier to achieve high scores, but unable to perfectly present the indicated motion from the reference video. Even in this case, our Dress&Dance still achieves very similar scores in all of the metrics, while the baseline, which achieves most of the best results, combines our try-on capability with Kling’s animation.

Ablation Study. We study the effectiveness of our training strategy, including the garment warm-up and multi-stage training. We define the “Direct Training (DT)” variant for Dress&Dance, which is directly trained with our final-stage inputs and outputs with full resolution. We compare full Dress&Dance with the DT-variant in Tab. 1 and project page. Without the garment warm-up training and multi-stage progressive training, the model is unable to faithfully preserve the contents of both user and garment images, even after extended training, resulting in low quantitative metrics. We observe that the training strategy is crucial for the convergence and final performance of the model.

5 Conclusion

We introduce a novel video diffusion framework, Dress&Dance, that enables both garment try-on and temporally consistent motion generation. As the first work to achieve high-resolution ($1152\times 720$) video virtual try-on, our framework creates high-quality videos showing the user wearing a target garment, with motion guided by an example video. Our approach tackles several key challenges, including preserving the likeness of the user and garment, and generating complex motion with minimal error propagation. By utilizing a unified conditioning network with cross-attention, we effectively handle heterogeneous inputs, improving garment registration, and supporting various garment capture methods. Additionally, our data-efficient training strategy, coupled with synthetic triplet data generation and a multi-stage progressive approach, enables the generation of high-resolution videos with reduced artifacts.