Attention Distillation: A Unified Approach to Visual Characteristics Transfer

TL;DR

This paper introduces a novel attention distillation method that leverages self-attention features from pretrained diffusion models to transfer visual styles, textures, and semantics to generated images, improving synthesis quality and speed.

Contribution

It proposes a new attention distillation loss and an improved classifier guidance method that integrate attention transfer into the diffusion process for enhanced image style transfer.

Findings

Effective transfer of style, appearance, and texture in image synthesis

Accelerated image generation through integrated guidance

Superior performance demonstrated in extensive experiments

Abstract

Recent advances in generative diffusion models have shown a notable inherent understanding of image style and semantics. In this paper, we leverage the self-attention features from pretrained diffusion networks to transfer the visual characteristics from a reference to generated images. Unlike previous work that uses these features as plug-and-play attributes, we propose a novel attention distillation loss calculated between the ideal and current stylization results, based on which we optimize the synthesized image via backpropagation in latent space. Next, we propose an improved Classifier Guidance that integrates attention distillation loss into the denoising sampling process, further accelerating the synthesis and enabling a broad range of image generation applications. Extensive experiments have demonstrated the extraordinary performance of our approach in transferring the examples' style, appearance, and texture to new images in synthesis. Code is available at https://github.com/xugao97/AttentionDistillation.