Pursuing Temporal-Consistent Video Virtual Try-On via Dynamic Pose Interaction

TL;DR

This paper introduces DPIDM, a diffusion-based framework that models dynamic pose interactions for video virtual try-on, significantly improving temporal consistency and visual authenticity over existing methods.

Contribution

The paper proposes a novel diffusion model with a skeleton-based pose adapter and hierarchical attention to effectively capture spatiotemporal human-garment pose interactions in video try-on.

Findings

Achieves a VFID score of 0.506 on VVT dataset, a 60.5% improvement over previous methods.

Outperforms baseline methods on multiple datasets in visual authenticity and temporal consistency.

Demonstrates the effectiveness of pose-aware spatial and temporal attention mechanisms.

Abstract

Video virtual try-on aims to seamlessly dress a subject in a video with a specific garment. The primary challenge involves preserving the visual authenticity of the garment while dynamically adapting to the pose and physique of the subject. While existing methods have predominantly focused on image-based virtual try-on, extending these techniques directly to videos often results in temporal inconsistencies. Most current video virtual try-on approaches alleviate this challenge by incorporating temporal modules, yet still overlook the critical spatiotemporal pose interactions between human and garment. Effective pose interactions in videos should not only consider spatial alignment between human and garment poses in each frame but also account for the temporal dynamics of human poses throughout the entire video. With such motivation, we propose a new framework, namely Dynamic Pose Interaction Diffusion Models (DPIDM), to leverage diffusion models to delve into dynamic pose interactions for video virtual try-on. Technically, DPIDM introduces a skeleton-based pose adapter to integrate synchronized human and garment poses into the denoising network. A hierarchical attention module is then exquisitely designed to model intra-frame human-garment pose interactions and long-term human pose dynamics across frames through pose-aware spatial and temporal attention mechanisms. Moreover, DPIDM capitalizes on a temporal regularized attention loss between consecutive frames to enhance temporal consistency. Extensive experiments conducted on VITON-HD, VVT and ViViD datasets demonstrate the superiority of our DPIDM against the baseline methods. Notably, DPIDM achieves VFID score of 0.506 on VVT dataset, leading to 60.5% improvement over the state-of-the-art GPD-VVTO approach.