SwiftTry: Fast and Consistent Video Virtual Try-On with Diffusion Models

TL;DR

SwiftTry introduces a fast, consistent video virtual try-on method using diffusion models, incorporating temporal attention and a novel caching technique to improve quality and efficiency, validated on a new challenging dataset.

Contribution

The paper presents a novel diffusion-based approach with temporal attention and ShiftCaching for efficient, consistent video virtual try-on, along with a new high-resolution dataset.

Findings

Outperforms existing methods in video consistency

Achieves faster inference speeds

Handles complex backgrounds and movements effectively

Abstract

Given an input video of a person and a new garment, the objective of this paper is to synthesize a new video where the person is wearing the specified garment while maintaining spatiotemporal consistency. Although significant advances have been made in image-based virtual try-on, extending these successes to video often leads to frame-to-frame inconsistencies. Some approaches have attempted to address this by increasing the overlap of frames across multiple video chunks, but this comes at a steep computational cost due to the repeated processing of the same frames, especially for long video sequences. To tackle these challenges, we reconceptualize video virtual try-on as a conditional video inpainting task, with garments serving as input conditions. Specifically, our approach enhances image diffusion models by incorporating temporal attention layers to improve temporal coherence. To reduce computational overhead, we propose ShiftCaching, a novel technique that maintains temporal consistency while minimizing redundant computations. Furthermore, we introduce the TikTokDress dataset, a new video try-on dataset featuring more complex backgrounds, challenging movements, and higher resolution compared to existing public datasets. Extensive experiments demonstrate that our approach outperforms current baselines, particularly in terms of video consistency and inference speed. The project page is available at https://swift-try.github.io/.