TED-VITON: Transformer-Empowered Diffusion Models for Virtual Try-On

TL;DR

TED-VITON introduces a transformer-based diffusion model for virtual try-on that significantly improves visual quality and text rendering accuracy by integrating specialized adapters, loss functions, and prompt optimization techniques.

Contribution

The paper presents TED-VITON, a novel framework that effectively leverages DiT-based T2I models for VTO, incorporating a Garment Semantic Adapter, Text Preservation Loss, and prompt optimization to enhance performance.

Findings

Achieved state-of-the-art visual quality in VTO images.

Significantly improved text rendering accuracy on garments.

Established new benchmarks for VTO tasks.

Abstract

Recent advancements in Virtual Try-On (VTO) have demonstrated exceptional efficacy in generating realistic images and preserving garment details, largely attributed to the robust generative capabilities of text-to-image (T2I) diffusion backbones. However, the T2I models that underpin these methods have become outdated, thereby limiting the potential for further improvement in VTO. Additionally, current methods face notable challenges in accurately rendering text on garments without distortion and preserving fine-grained details, such as textures and material fidelity. The emergence of Diffusion Transformer (DiT) based T2I models has showcased impressive performance and offers a promising opportunity for advancing VTO. Directly applying existing VTO techniques to transformer-based T2I models is ineffective due to substantial architectural differences, which hinder their ability to fully leverage the models' advanced capabilities for improved text generation. To address these challenges and unlock the full potential of DiT-based T2I models for VTO, we propose TED-VITON, a novel framework that integrates a Garment Semantic (GS) Adapter for enhancing garment-specific features, a Text Preservation Loss to ensure accurate and distortion-free text rendering, and a constraint mechanism to generate prompts by optimizing Large Language Model (LLM). These innovations enable state-of-the-art (SOTA) performance in visual quality and text fidelity, establishing a new benchmark for VTO task. Project page: https://zhenchenwan.github.io/TED-VITON/