UniVST: A Unified Framework for Training-free Localized Video Style Transfer

TL;DR

UniVST introduces a training-free, unified diffusion-based framework for localized video style transfer, enhancing temporal consistency and detail preservation without requiring model training.

Contribution

It proposes a novel training-free localized video style transfer method using diffusion models, with a point-matching mask propagation, AdaIN-guided stylization, and optical flow-based smoothing.

Findings

Outperforms existing methods in quantitative metrics

Achieves better temporal consistency and detail preservation

Operates without training, simplifying deployment

Abstract

This paper presents UniVST, a unified framework for localized video style transfer based on diffusion models. It operates without the need for training, offering a distinct advantage over existing diffusion methods that transfer style across entire videos. The endeavors of this paper comprise: (1) A point-matching mask propagation strategy that leverages the feature maps from the DDIM inversion. This streamlines the model's architecture by obviating the need for tracking models. (2) A training-free AdaIN-guided localized video stylization mechanism that operates at both the latent and attention levels. This balances content fidelity and style richness, mitigating the loss of localized details commonly associated with direct video stylization. (3) A sliding-window consistent smoothing scheme that harnesses optical flow within the pixel representation and refines predicted noise to update the latent space. This significantly enhances temporal consistency and diminishes artifacts in stylized video. Our proposed UniVST has been validated to be superior to existing methods in quantitative and qualitative metrics. It adeptly addresses the challenges of preserving the primary object's style while ensuring temporal consistency and detail preservation. Our code is available at https://github.com/QuanjianSong/UniVST.