E-MD3C: Taming Masked Diffusion Transformers for Efficient Zero-Shot Object Customization

TL;DR

E-MD3C introduces a lightweight, efficient masked diffusion transformer framework for zero-shot object image customization, significantly reducing computational resources while maintaining high-quality results.

Contribution

The paper presents a novel, resource-efficient masked diffusion transformer architecture with disentangled conditions and a learnable collector for improved zero-shot image customization.

Findings

Outperforms existing methods on VITON-HD dataset across multiple metrics.

Uses only 1/4 of the parameters and 2/3 of GPU memory compared to Unet-based models.

Achieves 2.5x faster inference speed with comparable or better quality.

Abstract

We propose E-MD3C ($\underline{E}$fficient $\underline{M}$asked $\underline{D}$iffusion Transformer with Disentangled $\underline{C}$onditions and $\underline{C}$ompact $\underline{C}$ollector), a highly efficient framework for zero-shot object image customization. Unlike prior works reliant on resource-intensive Unet architectures, our approach employs lightweight masked diffusion transformers operating on latent patches, offering significantly improved computational efficiency. The framework integrates three core components: (1) an efficient masked diffusion transformer for processing autoencoder latents, (2) a disentangled condition design that ensures compactness while preserving background alignment and fine details, and (3) a learnable Conditions Collector that consolidates multiple inputs into a compact representation for efficient denoising and learning. E-MD3C outperforms the existing approach on the VITON-HD dataset across metrics such as PSNR, FID, SSIM, and LPIPS, demonstrating clear advantages in parameters, memory efficiency, and inference speed. With only $\frac{1}{4}$ of the parameters, our Transformer-based 468M model delivers $2.5\times$ faster inference and uses $\frac{2}{3}$ of the GPU memory compared to an 1720M Unet-based latent diffusion model.