Tuning-Free Visual Customization via View Iterative Self-Attention Control

TL;DR

VisCtrl is a training-free, iterative self-attention method for visual customization that enables personalized editing of images, videos, and 3D scenes using only one reference image, without fine-tuning diffusion models.

Contribution

We introduce VisCtrl, a novel training-free approach that injects reference image features into target images via self-attention, eliminating the need for model fine-tuning.

Findings

Effective personalized editing with a single reference image.

Applicable across images, videos, and 3D scenes.

Outperforms fine-tuning methods in efficiency and flexibility.

Abstract

Fine-Tuning Diffusion Models enable a wide range of personalized generation and editing applications on diverse visual modalities. While Low-Rank Adaptation (LoRA) accelerates the fine-tuning process, it still requires multiple reference images and time-consuming training, which constrains its scalability for large-scale and real-time applications. In this paper, we propose \textit{View Iterative Self-Attention Control (VisCtrl)} to tackle this challenge. Specifically, VisCtrl is a training-free method that injects the appearance and structure of a user-specified subject into another subject in the target image, unlike previous approaches that require fine-tuning the model. Initially, we obtain the initial noise for both the reference and target images through DDIM inversion. Then, during the denoising phase, features from the reference image are injected into the target image via the self-attention mechanism. Notably, by iteratively performing this feature injection process, we ensure that the reference image features are gradually integrated into the target image. This approach results in consistent and harmonious editing with only one reference image in a few denoising steps. Moreover, benefiting from our plug-and-play architecture design and the proposed Feature Gradual Sampling strategy for multi-view editing, our method can be easily extended to edit in complex visual domains. Extensive experiments show the efficacy of VisCtrl across a spectrum of tasks, including personalized editing of images, videos, and 3D scenes.