Dress-1-to-3: Single Image to Simulation-Ready 3D Outfit with Diffusion Prior and Differentiable Physics

TL;DR

Dress-1-to-3 presents a novel pipeline for converting a single image into a simulation-ready, separable 3D garment model with sewing patterns, enabling realistic virtual try-on and dynamic garment animations.

Contribution

It introduces a new method combining sewing pattern generation, multi-view diffusion, and differentiable simulation to produce physics-plausible, separable 3D garments from a single image.

Findings

Enhanced geometric alignment of 3D garments with input images

Generated realistic dynamic garment animations

Produced simulation-ready, separable garments with sewing patterns

Abstract

Recent advances in large models have significantly advanced image-to-3D reconstruction. However, the generated models are often fused into a single piece, limiting their applicability in downstream tasks. This paper focuses on 3D garment generation, a key area for applications like virtual try-on with dynamic garment animations, which require garments to be separable and simulation-ready. We introduce Dress-1-to-3, a novel pipeline that reconstructs physics-plausible, simulation-ready separated garments with sewing patterns and humans from an in-the-wild image. Starting with the image, our approach combines a pre-trained image-to-sewing pattern generation model for creating coarse sewing patterns with a pre-trained multi-view diffusion model to produce multi-view images. The sewing pattern is further refined using a differentiable garment simulator based on the generated multi-view images. Versatile experiments demonstrate that our optimization approach substantially enhances the geometric alignment of the reconstructed 3D garments and humans with the input image. Furthermore, by integrating a texture generation module and a human motion generation module, we produce customized physics-plausible and realistic dynamic garment demonstrations. Project page: https://dress-1-to-3.github.io/