Significance of Skeleton-based Features in Virtual Try-On

TL;DR

This paper introduces a novel virtual try-on method that uses skeleton-based geometric features and independent part warping to improve pose robustness, especially for bent or crossed arms, without needing paired training data.

Contribution

It proposes a new approach combining handcrafted geometric features and learning modules to enhance pose-robust virtual try-on results, addressing limitations of existing warping methods.

Findings

Outperforms benchmark methods in pose-robustness

Effectively handles crossed arm and bent postures

Produces photo-realistic virtual try-on images

Abstract

The idea of \textit{Virtual Try-ON} (VTON) benefits e-retailing by giving an user the convenience of trying a clothing at the comfort of their home. In general, most of the existing VTON methods produce inconsistent results when a person posing with his arms folded i.e., bent or crossed, wants to try an outfit. The problem becomes severe in the case of long-sleeved outfits. As then, for crossed arm postures, overlap among different clothing parts might happen. The existing approaches, especially the warping-based methods employing \textit{Thin Plate Spline (TPS)} transform can not tackle such cases. To this end, we attempt a solution approach where the clothing from the source person is segmented into semantically meaningful parts and each part is warped independently to the shape of the person. To address the bending issue, we employ hand-crafted geometric features consistent with human body geometry for warping the source outfit. In addition, we propose two learning-based modules: a synthesizer network and a mask prediction network. All these together attempt to produce a photo-realistic, pose-robust VTON solution without requiring any paired training data. Comparison with some of the benchmark methods clearly establishes the effectiveness of the approach.