Diffuse to Choose: Enriching Image Conditioned Inpainting in Latent Diffusion Models for Virtual Try-All

TL;DR

Diffuse to Choose is a diffusion-based inpainting model that enables realistic virtual try-on by balancing detail preservation and semantic manipulation, offering fast inference and high fidelity in product visualization.

Contribution

The paper introduces a novel diffusion inpainting approach that integrates fine-grained reference features into latent space for improved virtual try-on applications.

Findings

Outperforms existing zero-shot diffusion inpainting methods.

Achieves better detail preservation compared to DreamPaint.

Demonstrates efficiency suitable for real-time applications.

Abstract

As online shopping is growing, the ability for buyers to virtually visualize products in their settings-a phenomenon we define as "Virtual Try-All"-has become crucial. Recent diffusion models inherently contain a world model, rendering them suitable for this task within an inpainting context. However, traditional image-conditioned diffusion models often fail to capture the fine-grained details of products. In contrast, personalization-driven models such as DreamPaint are good at preserving the item's details but they are not optimized for real-time applications. We present "Diffuse to Choose," a novel diffusion-based image-conditioned inpainting model that efficiently balances fast inference with the retention of high-fidelity details in a given reference item while ensuring accurate semantic manipulations in the given scene content. Our approach is based on incorporating fine-grained features from the reference image directly into the latent feature maps of the main diffusion model, alongside with a perceptual loss to further preserve the reference item's details. We conduct extensive testing on both in-house and publicly available datasets, and show that Diffuse to Choose is superior to existing zero-shot diffusion inpainting methods as well as few-shot diffusion personalization algorithms like DreamPaint.