OmniDiT: Extending Diffusion Transformer to Omni-VTON Framework

TL;DR

OmniDiT is a unified diffusion transformer framework for virtual try-on that improves detail preservation, scene generalization, and inference efficiency by integrating multiple techniques and a large, diverse dataset.

Contribution

The paper introduces OmniDiT, a novel diffusion transformer model that unifies try-on and try-off tasks, incorporating a new dataset, adaptive encoding, and shifted window attention for improved performance.

Findings

Achieves state-of-the-art results in model-free VTON and VTOFF tasks.

Performs comparably to SOTA in model-based VTON.

Demonstrates effective handling of complex scenes and detailed garment fitting.

Abstract

Despite the rapid advancement of Virtual Try-On (VTON) and Try-Off (VTOFF) technologies, existing VTON methods face challenges with fine-grained detail preservation, generalization to complex scenes, complicated pipeline, and efficient inference. To tackle these problems, we propose OmniDiT, an omni Virtual Try-On framework based on the Diffusion Transformer, which combines try-on and try-off tasks into one unified model. Specifically, we first establish a self-evolving data curation pipeline to continuously produce data, and construct a large VTON dataset Omni-TryOn, which contains over 380k diverse and high-quality garment-model-tryon image pairs and detailed text prompts. Then, we employ the token concatenation and design an adaptive position encoding to effectively incorporate multiple reference conditions. To relieve the bottleneck of long sequence computation, we are the first to introduce Shifted Window Attention into the diffusion model, thus achieving a linear complexity. To remedy the performance degradation caused by local window attention, we utilize multiple timestep prediction and an alignment loss to improve generation fidelity. Experiments reveal that, under various complex scenes, our method achieves the best performance in both the model-free VTON and VTOFF tasks and a performance comparable to current SOTA methods in the model-based VTON task.