SIFT-VTON: Geometric Correspondence Supervision on Cross-Attention for Virtual Try-On

TL;DR

SIFT-VTON introduces explicit geometric supervision using SIFT keypoints to improve the accuracy and detail preservation in diffusion-based virtual try-on, outperforming previous implicit methods.

Contribution

The paper presents a novel method that integrates SIFT keypoint matching as explicit geometric guidance into diffusion models for virtual try-on, enhancing detail preservation and alignment.

Findings

Significant improvements on unpaired metrics in VITON-HD dataset

Better preservation of text and pattern details in qualitative results

Focused attention on relevant garment regions confirmed by visualization

Abstract

Diffusion-based virtual try-on methods achieve photorealistic synthesis through cross-attention mechanisms that transfer garment features to target body regions. However, these approaches rely on implicit learning of spatial correspondences, struggling to preserve fine details such as text and illustrations. We propose a novel approach, which we call SIFT-VTON, that utilizes SIFT keypoint matching to provide explicit geometric guidance for diffusion-based virtual try-on. Our method applies domain-specific filtering to SIFT keypoint matches between garment and person images, then converts these correspondences into spatial probability distributions that supervise cross-attention layers during training. This explicit supervision guides the model to learn precise spatial alignment, concentrating attention on geometrically consistent garment regions. Experiments on the VITON-HD dataset demonstrate significant improvements on unpaired metrics while maintaining competitive paired reconstruction metrics. Qualitative comparisons show superior preservation of text clarity and pattern alignment. Attention visualizations confirm that our method produces sharply focused attention on relevant garment details. This work demonstrates that classical geometric correspondence methods can effectively enhance modern diffusion models for conditional synthesis tasks. The source code will be available at https://github.com/takesukeDS/SIFT-VTON.