MMTryon: Multi-Modal Multi-Reference Control for High-Quality Fashion Generation

TL;DR

MMTryon is a novel multi-modal, multi-reference virtual try-on framework that generates high-quality, style-controllable fashion images without relying on segmentation, supporting multiple items and diverse dressing styles.

Contribution

It introduces a multi-modality, multi-reference attention mechanism and a segmentation-free training pipeline, enabling multi-item, style-controllable virtual try-on without segmentation dependency.

Findings

Outperforms state-of-the-art methods qualitatively and quantitatively.

Supports multiple try-on items and customizable dressing styles.

Operates effectively on high-resolution and in-the-wild images.

Abstract

This paper introduces MMTryon, a multi-modal multi-reference VIrtual Try-ON (VITON) framework, which can generate high-quality compositional try-on results by taking a text instruction and multiple garment images as inputs. Our MMTryon addresses three problems overlooked in prior literature: 1) Support of multiple try-on items. Existing methods are commonly designed for single-item try-on tasks (e.g., upper/lower garments, dresses). 2)Specification of dressing style. Existing methods are unable to customize dressing styles based on instructions (e.g., zipped/unzipped, tuck-in/tuck-out, etc.) 3) Segmentation Dependency. They further heavily rely on category-specific segmentation models to identify the replacement regions, with segmentation errors directly leading to significant artifacts in the try-on results. To address the first two issues, our MMTryon introduces a novel multi-modality and multi-reference attention mechanism to combine the garment information from reference images and dressing-style information from text instructions. Besides, to remove the segmentation dependency, MMTryon uses a parsing-free garment encoder and leverages a novel scalable data generation pipeline to convert existing VITON datasets to a form that allows MMTryon to be trained without requiring any explicit segmentation. Extensive experiments on high-resolution benchmarks and in-the-wild test sets demonstrate MMTryon's superiority over existing SOTA methods both qualitatively and quantitatively. MMTryon's impressive performance on multi-item and style-controllable virtual try-on scenarios and its ability to try on any outfit in a large variety of scenarios from any source image, opens up a new avenue for future investigation in the fashion community.