Harnessing the Latent Diffusion Model for Training-Free Image Style Transfer

TL;DR

This paper introduces a training-free style transfer method using latent diffusion models, enabling quick and compatible style transfer without additional training by tracking style encoding during the reverse diffusion process.

Contribution

The paper presents STRDP, a novel training-free style transfer algorithm that leverages AdaIN during reverse diffusion in LDMs, reducing computational costs and enhancing compatibility.

Findings

Effective style transfer without retraining.

Fast processing suitable for real-time applications.

High user satisfaction in qualitative assessments.

Abstract

Diffusion models have recently shown the ability to generate high-quality images. However, controlling its generation process still poses challenges. The image style transfer task is one of those challenges that transfers the visual attributes of a style image to another content image. Typical obstacle of this task is the requirement of additional training of a pre-trained model. We propose a training-free style transfer algorithm, Style Tracking Reverse Diffusion Process (STRDP) for a pretrained Latent Diffusion Model (LDM). Our algorithm employs Adaptive Instance Normalization (AdaIN) function in a distinct manner during the reverse diffusion process of an LDM while tracking the encoding history of the style image. This algorithm enables style transfer in the latent space of LDM for reduced computational cost, and provides compatibility for various LDM models. Through a series of experiments and a user study, we show that our method can quickly transfer the style of an image without additional training. The speed, compatibility, and training-free aspect of our algorithm facilitates agile experiments with combinations of styles and LDMs for extensive application.