Q-Align: Alleviating Attention Leakage in Zero-Shot Appearance Transfer via Query-Query Alignment

TL;DR

Q-Align introduces a novel Query-Query alignment method to reduce attention leakage in zero-shot appearance transfer, significantly improving semantic alignment and appearance fidelity in large-scale image generation models.

Contribution

The paper proposes Q-Align, a new approach that uses Query-Query alignment, key-value rearrangement, and attention refinement to enhance zero-shot appearance transfer.

Findings

Q-Align outperforms state-of-the-art methods in appearance fidelity.

It effectively mitigates attention leakage in large-scale image generation.

Q-Align maintains competitive structure preservation.

Abstract

We observe that zero-shot appearance transfer with large-scale image generation models faces a significant challenge: Attention Leakage. This challenge arises when the semantic mapping between two images is captured by the Query-Key alignment. To tackle this issue, we introduce Q-Align, utilizing Query-Query alignment to mitigate attention leakage and improve the semantic alignment in zero-shot appearance transfer. Q-Align incorporates three core contributions: (1) Query-Query alignment, facilitating the sophisticated spatial semantic mapping between two images; (2) Key-Value rearrangement, enhancing feature correspondence through realignment; and (3) Attention refinement using rearranged keys and values to maintain semantic consistency. We validate the effectiveness of Q-Align through extensive experiments and analysis, and Q-Align outperforms state-of-the-art methods in appearance fidelity while maintaining competitive structure preservation.

Tables

Table 1. Table 1: Mean and standard deviation of Gram Loss and IoU for each domain (Mean / Std).

Domain	Gram Loss$\downarrow$		IoU$\uparrow$
	Cross-Image	Q-Align	Cross-Image	Q-Align
Animal	1.08 / 0.64	1.68 / 1.77	0.66 / 0.16	0.71 / 0.12
Animal Face	0.79 / 0.43	0.84 / 0.42	0.72 / 0.18	0.78 / 0.10
Bird	1.47 / 0.85	1.37 / 0.75	0.75 / 0.15	0.80 / 0.09
Building	4.92 / 4.03	4.68 / 2.38	0.81 / 0.17	0.82 / 0.17
Cake	3.16 / 3.26	2.58 / 2.11	0.78 / 0.23	0.84 / 0.11
Car	1.12 / 0.32	1.32 / 0.35	0.84 / 0.17	0.83 / 0.17
Avg	2.09 / 1.59	2.08 / 1.30	0.76 / 0.18	0.80 / 0.13

Table 2. Table 2: GPT-based appearance fidelity score across different methods (Mean/Std).

Appearance Fidelity Score
Domain	SpliceViT	DiffuseIT	DragonDiffusion	Cross-Image	Q-Align (ours)
Animal	2.54 / 0.51	3.65 / 0.47	3.69 / 0.58	4.04 / 0.50	4.25 / 0.41
Animal Face	2.49 / 0.58	3.41 / 0.39	3.72 / 0.53	4.29 / 0.28	4.31 / 0.30
Bird	2.68 / 0.39	3.36 / 0.48	3.88 / 0.53	4.15 / 0.29	4.31 / 0.19
Building	2.16 / 0.50	3.20 / 0.39	3.52 / 0.35	4.36 / 0.14	4.46 / 0.17
Cake	2.56 / 0.64	2.93 / 0.49	4.15 / 0.51	4.09 / 0.42	4.23 / 0.32
Car	2.23 / 0.40	3.21 / 0.40	3.86 / 0.41	4.22 / 0.36	4.21 / 0.42
Avg	2.44 / 0.54	3.29 / 0.49	3.80 / 0.53	4.19 / 0.37	4.30 / 0.33

Equations

$$\text{softmax}\left(\frac{Q_{\text{out}}{K}_{\text{app}}^{\top }}{\sqrt{d}}\right)V_{\text{app}}.$$

$$S=Q_{\text{app}}{Q}_{\text{str}}^{\top },$$

$$(P^{\top }{)}_{r,c}=\{\begin{array}{ll}\frac{1}{k},& \text{if }{S}_{r,c}\in \text{Top-}k(S_r)\\ 0,& \text{otherwise}\end{array}$$

$$P_{r,r}=1,\text{ If }\sum _{c=1}^n{P}_{r,c}=0.$$

$$P_{r,c}^{\prime }=\{\begin{array}{ll}\frac{e^{P_{r,c}}}{{\sum }_{c^{\prime }\in [n],P_{r,c^{\prime }}\ne 0}{e}^{P_{r,c^{\prime }}}},& \text{if }{P}_{r,c}\ne 0\\ 0,& \text{if }{P}_{r,c}=0\end{array}$$

$$K^{\ast }$$

$$V^{\ast }$$

$$\text{softmax}\left(\frac{Q_{\text{out}}(K^{\ast }{)}^{\top }}{\sqrt{d}}\right)V^{\ast }.$$

Full text

††footnotetext: * These authors contributed equally to this work.††footnotetext: † Corresponding author.

1 Introduction

Appearance transfer is an image generation process where the structure image establishes the framework of semantic regions, and the appearance image provides the texture and identity details to fill those regions. As illustrated in Figure 1, for example, the wings in the appearance image are transferred to the wing regions of the structure image, branches to branches, and background to background. To achieve this, the primary challenge is achieving precise semantic alignment between the features of the two images.

Early methods relied on rule-based techniques [Lowe(2004), Bay et al.(2006)Bay, Tuytelaars, and Van Gool] for semantic correspondence, but were limited in their adaptability to diverse and complex datasets. With advances in deep learning, data-driven approaches [Lee et al.(2021)Lee, DeGol, Fragoso, and Sinha, Cho et al.(2023)Cho, Hong, and Kim, Truong et al.(2020)Truong, Danelljan, Gool, and Timofte, Teed and Deng(2020)] were introduced to improve alignment accuracy. Although these methods demonstrated promising performance, they depend on ground-truth correspondence and require extensive labeled training datasets. Pre-trained vision transformer features [Caron et al.(2021)Caron, Touvron, Misra, Jégou, Mairal, Bojanowski, and Joulin, Gupta et al.(2023)Gupta, Jampani, Esteves, Shrivastava, Makadia, Snavely, and Kar] have gained attention as a solution to overcome these limitations by enabling the extraction of correspondence maps without labeled datasets [Hamilton et al.(2022)Hamilton, Zhang, Hariharan, Snavely, and Freeman, Tumanyan et al.(2022)Tumanyan, Bar-Tal, Bagon, and Dekel].

More recently, large-scale image generation models [Rombach et al.(2022)Rombach, Blattmann, Lorenz, Esser, and Ommer, Gu et al.(2022)Gu, Chen, Bao, Wen, Zhang, Chen, Yuan, and Guo, Ramesh et al.(2022)Ramesh, Dhariwal, Nichol, Chu, and Chen] have introduced powerful generative capabilities that can be leveraged to discover semantic maps [Tang et al.(2023)Tang, Jia, Wang, Phoo, and Hariharan, Luo et al.(2023)Luo, Dunlap, Park, Holynski, and Darrell, Hedlin et al.(2023)Hedlin, Sharma, Mahajan, Isack, Kar, Tagliasacchi, and Yi, Zhang et al.(2023)Zhang, Herrmann, Hur, Polania Cabrera, Jampani, Sun, and Yang] in a zero-shot manner. These models are utilized in various ways, such as using the generative model as an encoder to extract semantically aligned features [Tang et al.(2023)Tang, Jia, Wang, Phoo, and Hariharan, Luo et al.(2023)Luo, Dunlap, Park, Holynski, and Darrell, Zhang et al.(2023)Zhang, Herrmann, Hur, Polania Cabrera, Jampani, Sun, and Yang] or through attention control [Cao et al.(2023)Cao, Wang, Qi, Shan, Qie, and Zheng, Hertz et al.(2023)Hertz, Mokady, Tenenbaum, Aberman, Pritch, and Cohen-Or, Parmar et al.(2023)Parmar, Kumar Singh, Zhang, Li, Lu, and Zhu, Tumanyan et al.(2023)Tumanyan, Geyer, Bagon, and Dekel] to manipulate specific regions. Further research has analyzed the role of attention [Liu et al.(2024)Liu, Wang, Cao, Jia, and Huang] and optimized its mechanisms [Wang et al.(2023)Wang, Yang, Yang, Butt, and van de Weijer] for improved performance.

The prominent method in zero-shot appearance transfer, Cross-Image [Alaluf et al.(2024)Alaluf, Garibi, Patashnik, Averbuch-Elor, and Cohen-Or], effectively performs appearance transfer without training by leveraging large-scale image generation models and employing attention control, based on MasaCtrl [Cao et al.(2023)Cao, Wang, Qi, Shan, Qie, and Zheng]. Despite its effectiveness, Cross-Image exhibits a significant challenge: attention leakage. We define the attention leakage as instances where the object(or background) is mapped to the background(or object), or the semantic mapping within the object itself is incorrect. As shown in Figure 1, attention to the bird’s body spills over to random background regions rather than remaining focused on the bird itself.

This paper proposes Q-Align, a novel approach for zero-shot appearance transfer that mitigates attention leakage without requiring any optimization process or additional data. We provide an in-depth analysis of attention leakage in Cross-Image and introduce a novel approach based on query-query alignment, as follows:

• •

We highlight our key observation that naive mixing of query, key, and value can cause misalignment between features, leading to attention leakage (Section 3).

• •

We demonstrate that query-query alignment offers more sophisticated spatial semantic mapping, and propose a key-value rearrangement technique to improve query-key-value correspondence in the conventional Cross-Image attention approach (Section 4).

• •

We validate the effectiveness of the proposed method through a novel evaluation protocol leveraging GPT-4o series, and provide a detailed analysis of each component (Section 5).

The official implementation of Q-Align is publicly available111https://github.com/SEED-TO-TREE/Q-Align-BMVC2025.

2 Preliminary

Problem Formulation. Given a pair of images, $I_{\text{app}}$ and $I_{\text{str}}$, the appearance transfer task aims to generate $I_{\text{out}}$ by transferring the appearance of $I_{\text{app}}$ onto $I_{\text{str}}$, mapping semantically corresponding regions and ensuring coherence between objects and backgrounds. In Figure 1, features such as wings, beak, and branches in $I_{\text{app}}$ should be aligned with semantically corresponding regions in $I_{\text{str}}$. Additionally, the background of $I_{\text{app}}$ needs to be smoothly integrated into that of $I_{\text{str}}$, creating a cohesive and visually seamless result.

Cross-Image Attention. The cross-image attention mechanism [Alaluf et al.(2024)Alaluf, Garibi, Patashnik, Averbuch-Elor, and Cohen-Or] is built upon the mutual attention of MasaCtrl [Cao et al.(2023)Cao, Wang, Qi, Shan, Qie, and Zheng], which demonstrates how the appearance of input image can be semantically mapped to the structure of the output image. Since self-attention layers allow the model to capture feature correspondences within an image, the queries, keys, and values of these layers can also be leveraged to infer corresponding regions between different images. Let $(Q_{\text{app}},K_{\text{app}},V_{\text{app}})$ and $(Q_{\text{out}},K_{\text{out}},V_{\text{out}})$ denote the queries, keys, and values corresponding to $I_{\text{app}}$ and $I_{\text{out}}$, respectively, within a specific self-attention layer. The cross-image attention is defined by replacing $K_{\text{out}},V_{\text{out}}$ with $K_{\text{app}},V_{\text{app}}$ as follows:

[TABLE]

$Q_{\text{out}}$ determines the spatial semantics and is initialized as $Q_{\text{str}}$ at the first diffusion step. $K_{\text{app}}$ offers appearance context for each query. By doing so, Cross-Image enables the model to implicitly transfer the visual appearance between semantically similar objects in the different images.

3 Motivation

Attention Leakage. Figure 1 shows attention leakage in Cross-Image during the appearance transfer from $I_{\text{app}}$ to $I_{\text{str}}$, causing unreliable generation in $I_{\text{Cross-Image}}$. To analyze the artifact in the upper left of $I_{\text{Cross-Image}}$, we examine the attention map for the bird’s body (green square) in $I_{\text{app}}$. The attention erroneously shifts to a background region instead of focusing on the bird (i.e., it leaks into irrelevant areas). This misaligns the background and draws attention to unrelated features (e.g\bmvaOneDot, a mint-colored bird). In our experiment, attention leakage occurs in 94 of 180 pairs (52.2%), and Q-Align improves 65 of these (69.1%).

Query-Key Misalignment. Cross-Image relies solely on $Q_{\text{out}}K_{\text{app}}^{\top}$ to infer semantically corresponding regions between $I_{\text{out}}$ and $I_{\text{app}}$, and faces the attention leakage. This issue arises when the attention mechanism fails to accurately attend to the corresponding regions, shifting instead to incorrect locations and generating erroneous images. We will discuss further on Query-Key misalignment with Figure 3 in the following section.

Naive Approaches. Cross-Image introduces an attention contrasting hyperparameter to promote relevant alignment and suppress noise. One might expect tuning it could fix leakage, but as shown in Figure 3, the issue persists without proper Query-Key alignment. Another strategy is to apply an object mask [Mou et al.(2024)Mou, Wang, Song, Shan, and Zhang], yet leakage can still occur within the masked area. Even with accurate masks, the model often overfits local style rather than ensuring semantic alignment, resulting in unnatural images. We revisit this in the experiments (Figure 6).

4 Q-Align

We propose Q-Align to address misalignment between semantically corresponding regions in the image pair. Specifically, we obtain sophisticated feature mapping between two images based on query-query alignment, and rearrange key and value for better query-key-value correspondence. Lastly, refined attention is created by utilizing the rearranged keys and values, enabling the capture of semantic correspondence between both images. The entire procedure of Q-Align is illustrated in Figure 4.

4.1 Query-Query alignment

The main cause of attention leakage is the feature misalignment in the cross-image attention. To mitigate this, we leverage the query-query alignment between $Q_{\text{str}}$ and $Q_{\text{app}}$. Since queries determine the semantic meaning of each spatial location in an image, we claim that query-query alignment can provide aligned features between two images more accurately than query-key alignment of Cross-Image. In this paper, we measure the query-query alignment with an alignment matrix $S$:

[TABLE]

where $S\in\mathbb{R}^{n\times n}$ and $n$ is the number of queries. Each element $S_{r,c}$ indicates the similarity score between the $r$-th query of $Q_{\text{app}}$ and the $c$-th query of $Q_{\text{str}}$.

Empirical Evidence. Figure 3 shows that the query-query alignment ($\text{softmax}(Q_{\text{str}}Q_{\text{app}}^{\top})$) achieves better correspondence than query-key alignment ($\text{softmax}(Q_{\text{str}}K_{\text{app}}^{\top})$). The query-query alignment provides more sophisticated spatial attention, while the query-key alignment distributes attention broadly across related regions. For example, for the finial tip in $I_{\text{app}}$ (second column in the green grid), query-key alignment distributes attentions to the body part of the building in $I_{\text{str}}$, while our query-query alignment sharply focuses on the finial tip in $I_{\text{str}}$. Similarly, for the right pitched roof in $I_{\text{app}}$ (third column in the green grid), query-key alignment broadly focuses on the background in $I_{\text{str}}$, whereas our query-query alignment precisely attends to the right pitched roof area in $I_{\text{str}}$. Therefore, by leveraging the query-query alignment to extract precise and detailed points of interest, we could enable the model to accurately and comprehensively focus on relevant regions.

4.2 Key-Value Rearrangement

We utilize the query-query alignment $S$ between $Q_{\text{str}}$ and $Q_{\text{app}}$ to rearrange $(K_{\text{app}},V_{\text{app}})$ for better query-key-value correspondence, based on our demonstration that $S$ provides precise feature alignment between $I_{\text{str}}$ and $I_{\text{app}}$. It is important to recognize that we modify the keys and values rather than the queries, as queries define the semantic meaning of each spatial location in an image. Modifying the queries could potentially disrupt the structural semantics of the output image.

Rearrangement Matrix. Specifically, we construct an aggregation matrix $P\in\mathbb{R}^{n\times n}$ for the key-value rearrangement based on the query-query alignment $S$, as follows:

[TABLE]

$\text{Top-}k(S_{r})$ denotes the set of top-$k$ values in each row $S_{r}\in\mathbb{R}^{n}$. It is worth noting that we only retain top-$k$ alignments for key-value rearrangement, as larger $k$ broadens the range of features aggregated and may dilute the distinctiveness of the generated image. In this paper, we use $k=1$. If the sum of any row in $P$ is 0, we set the diagonal entry to 1 to maintain the original keys/values:

[TABLE]

We then apply row-wise softmax reweighting to $P$:

[TABLE]

It is noted that we ensure that elements with an initial value of 0 remain unchanged. Each element $P^{\prime}_{r,c}$ indicates the contribution weights of the $c$-th feature on the $r$-th feature.

Key-Value Rearrangement via Aggregation. The aggregation matrix $P^{\prime}$ is applied to modify the original key matrix $K_{\text{app}}$ and the original value matrix $V_{\text{app}}$, to create a rearranged key matrix $K^{*}$ and value matrix $V^{*}$:

[TABLE]

For example, if $P^{\prime}_{10,20}=1$, the 20th key and value are rearranged to be aligned with the 10th query. By leveraging the query-query alignment to rearrange the key/value matrix, $K^{*}$ and $V^{*}$ achieve improved feature alignment with $Q_{\text{str}}$. This alignment extends to $Q_{\text{out}}$ used for generating the output image. Consequently, the modified key $K^{*}$ and the modified value $V^{*}$ align their features with the most relevant query vectors, thereby mitigating the issue of attention leakage.

Rearranged Cross-Image Attention Finally, we deploy the rearranged key $K^{*}$ and value $V^{*}$ for the Cross-Image attention between $I_{\text{out}}$ and $I_{\text{app}}$:

[TABLE]

This formulation maintains semantic consistency during the appearance transfer, ensuring that $Q_{\text{out}}$, $K^{*}$, and $V^{*}$ are aligned correctly while reducing the generation of unintended artifacts.

5 Experiments

5.1 Experiment setting

Datasets. Following Cross-Image [Alaluf et al.(2024)Alaluf, Garibi, Patashnik, Averbuch-Elor, and Cohen-Or], we selected six domains (animal faces, animals, cars, birds, buildings, and cakes) to enable a fair comparison. As done in Cross-Image [Alaluf et al.(2024)Alaluf, Garibi, Patashnik, Averbuch-Elor, and Cohen-Or], for the animal face domain, images from AFHQ [Choi et al.(2020)Choi, Uh, Yoo, and Ha] were utilized. For the remaining domains, we utilized images from the official Cross-Image page as well as copyright-free real-world images from Unsplash222https://unsplash.com. To ensure diversity in the dataset, we gathered 30 image pairs per domain, resulting in a total of 180 pairs.

Baseline Methods. We compare Q-Align with state-of-the-art approaches, including Cross-Image [Alaluf et al.(2024)Alaluf, Garibi, Patashnik, Averbuch-Elor, and Cohen-Or], DragonDiffusion [Mou et al.(2024)Mou, Wang, Song, Shan, and Zhang], DiffuseIT [Kwon and Ye(2023)], and SpliceViT [Tumanyan et al.(2022)Tumanyan, Bar-Tal, Bagon, and Dekel]. All methods were implemented using the official code provided by the respective authors, following their default parameters. We use Stable Diffusion [Rombach et al.(2022)Rombach, Blattmann, Lorenz, Esser, and Ommer] as the base model, and the same random seed and other processing parameters are applied uniformly. Further details on parameter settings and implementation specifics are provided in the supplementary materials.

Evaluation Metrics (existing). In the earlier work [Alaluf et al.(2024)Alaluf, Garibi, Patashnik, Averbuch-Elor, and Cohen-Or], they used Gram loss [Gatys et al.(2015)Gatys, Ecker, and Bethge] computed on VGG-19 features [Simonyan and Zisserman(2015)] for the appearance fidelity, and IoU between object masks [Ke et al.(2023)Ke, Ye, Danelljan, Liu, Tai, Tang, and Yu] of $I_{\text{out}}$ and $I_{\text{str}}$ for the structural consistency. However, as shown in Table 1, applying those evaluation metrics proposed in Cross-Image led to significant variance across different samples, which limited the reliability of the quantitative assessment. Furthermore, Figure 5 shows that the appearance similarity perceived by humans does not align well with the results of the Gram loss metric.

GPT-based Metrics (proposed). To address these limitations, we propose a novel evaluation protocol leveraging GPT-4o series [Hurst et al.(2024)Hurst, Lerer, Goucher, Perelman, Ramesh, Clark, Ostrow, Welihinda, Hayes, Radford, et al.]333we use gpt-4o-mini-2024-07-18 with OpenAI API, a powerful large language model with a vision capability. Inspired by [Liu et al.(2023)Liu, Iter, Xu, Wang, Xu, and Zhu], we prompt GPT with a detailed explanation of the appearance transfer task and explicitly defined two evaluation criteria (i.e., appearance fidelity and structural consistency). We then instruct GPT to rate each criterion on a scale from 1.0 to 5.0. A comprehensive description of the prompt design is provided in the supplementary material. As shown in Figure 5, GPT assigns scores at a level comparable to human evaluation while also providing explanations for its reasoning.

5.2 Experiment results

Quantitative Comparison. Table 3 and Table 3 present the quantitative evaluation of appearance transfer across different methods, based on the proposed GPT-based appearance fidelity and structural consistency scores. In terms of appearance, Q-Align achieved the best results in most domains, with the highest average score. In terms of structure, Q-Align also achieved the highest average structural consistency score, surpassing other methods by notable margins. By addressing the attention leakage issue and preventing unnecessary artifacts, Q-Align achieved both strong structure preservation and high appearance similarity, indicating that it delivered the best overall results for the appearance transfer task.

Qualitative Results. Figure 6 presents a comparison of appearance transfer results across different models, highlighting the distinct advantages of Q-Align. DragonDiffusion showed vibrant color but produced distorted images. As a mask-based approach, it prioritized fitting the style within the mask region rather than ensuring accurate semantic mapping. For instance, in the building example, the appearance image contained rounded roofs, meaning that the appearance transfer output should exhibit a rounded roof shape. However, DragonDiffusion forced the style into the original pointed roof mask, resulting in an unnatural composition. Cross-Image exhibited attention leakage, where objects appeared in the background, background elements were incorrectly mapped onto objects, or incorrect semantic mappings occurred within both object and background regions. For example, in the bird image, wing characteristics appeared in the background, and in the vehicle image, bumper traits (e.g., gray tones) were mapped onto the doors. Additionally, in the animal category, the background color was incorrectly mapped. In contrast, Q-Align demonstrated reliable performance in preserving both structural and appearance features, consistently generating high-quality and faithful appearance transfer results across diverse examples.

5.3 Study on Q-Align

Figure 7 shows the appearance transfer results of Cross-Image and Q-Align. We first observe that Cross-Image generates an undesirable image of a chair. A clue for this issue can be found in the Query-Key alignment, where the background is incorrectly aligned to the center of the chair. As a result, Cross-Image generates a background-colored hole in the chair. Query-Query alignment, on the other hand, prevents any attention leakage from the background onto the chair. The attention difference map $M_{\text{diff}}$ highlights areas with significant differences in semantic focus, emphasizing central regions where attention leakage occurs in Cross-Image. We observe that, when generating the center of the chair, Cross-Image attends to both the chair and the background, whereas Q-Align focuses solely on the chair itself. This result aligns with Zhang et al. [Zhang et al.(2021)Zhang, FAN, Zheng, Tanwisuth, and Zhou], who highlighted that queries and keys reside in distinct embedding spaces. We assert that query-query alignment, occurring within the same embedding space, inherently achieves better alignment than query-key alignment.

6 Related Work

Semantic Alignment. Most studies have focused on learning semantic correspondences between images through supervised training [Lee et al.(2021)Lee, DeGol, Fragoso, and Sinha, Cho et al.(2023)Cho, Hong, and Kim, Truong et al.(2020)Truong, Danelljan, Gool, and Timofte, Teed and Deng(2020)]. While these approaches demonstrate high performance, they come with the limitation of requiring large-scale datasets. To address this, methods that utilize features from pre-trained vision transformers [Caron et al.(2021)Caron, Touvron, Misra, Jégou, Mairal, Bojanowski, and Joulin, Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby, Gupta et al.(2023)Gupta, Jampani, Esteves, Shrivastava, Makadia, Snavely, and Kar] have been proposed, along with the image manipulation methodologies [Tumanyan et al.(2022)Tumanyan, Bar-Tal, Bagon, and Dekel, Kwon and Ye(2023)]. More recent approaches utilize large-scale image generation models [Rombach et al.(2022)Rombach, Blattmann, Lorenz, Esser, and Ommer, Gu et al.(2022)Gu, Chen, Bao, Wen, Zhang, Chen, Yuan, and Guo, Ramesh et al.(2022)Ramesh, Dhariwal, Nichol, Chu, and Chen] to extract features that enable precise semantic mapping between images [Tang et al.(2023)Tang, Jia, Wang, Phoo, and Hariharan, Luo et al.(2023)Luo, Dunlap, Park, Holynski, and Darrell, Zhang et al.(2023)Zhang, Herrmann, Hur, Polania Cabrera, Jampani, Sun, and Yang, Hedlin et al.(2023)Hedlin, Sharma, Mahajan, Isack, Kar, Tagliasacchi, and Yi]. Additionally, controlling the attention map of large models for manipulation has been explored [Cao et al.(2023)Cao, Wang, Qi, Shan, Qie, and Zheng, Hertz et al.(2023)Hertz, Mokady, Tenenbaum, Aberman, Pritch, and Cohen-Or, Parmar et al.(2023)Parmar, Kumar Singh, Zhang, Li, Lu, and Zhu, Tumanyan et al.(2023)Tumanyan, Geyer, Bagon, and Dekel], enabling semantic coherence to be preserved without the need for additional training.

Attention in Diffusion Models. Attention mechanisms [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] are fundamental in diffusion-based image manipulation, with numerous studies focused on optimizing [Wang et al.(2023)Wang, Yang, Yang, Butt, and van de Weijer], modifying [Cao et al.(2023)Cao, Wang, Qi, Shan, Qie, and Zheng, Xu et al.(2024)Xu, Huang, Pan, Ma, and Chai, Koo et al.(2024)Koo, Yoon, Hong, and Yoo, Hertz et al.(2023)Hertz, Mokady, Tenenbaum, Aberman, Pritch, and Cohen-Or, Parmar et al.(2023)Parmar, Kumar Singh, Zhang, Li, Lu, and Zhu], and refining [Li et al.(2024)Li, Zeng, Feng, Gao, Liu, Liu, Li, Tang, Hu, Liu, and Zhang, Couairon et al.(2023)Couairon, Verbeek, Schwenk, and Cord]. Among these, methods that mix the query of one image with the key and value of another have been developed to support non-rigid image editing [Cao et al.(2023)Cao, Wang, Qi, Shan, Qie, and Zheng, Xu et al.(2024)Xu, Huang, Pan, Ma, and Chai, Koo et al.(2024)Koo, Yoon, Hong, and Yoo]. Other studies utilize refined attention masks to guide the accurate designation of regions for modification within an image [Li et al.(2024)Li, Zeng, Feng, Gao, Liu, Liu, Li, Tang, Hu, Liu, and Zhang, Couairon et al.(2023)Couairon, Verbeek, Schwenk, and Cord], while some employ latent optimization to prevent focusing on irrelevant regions [Wang et al.(2023)Wang, Yang, Yang, Butt, and van de Weijer]. In this study, we propose a zero-shot method that operates without any external masks or additional optimization, relying solely on query-query alignment. This design effectively mitigates attention leakage while preserving both structural consistency and semantic fidelity unlike mask-based approaches that distort semantics by forcing styles into fixed mask regions.

7 Conclusion

We proposed Q-Align, a novel method that utilizes Query-Query alignment to effectively rearrange keys and values for the attention mechanism. Our method offers a zero-shot, straightforward solution that directly addresses the critical issue of attention leakage observed in SOTA methods. Through comprehensive analysis and extensive experiments, we demonstrated that Q-Align not only effectively resolves attention leakage but also significantly outperforms existing approaches. As future work, we envision extending Q-Align to incorporate fine-grained semantic mapping techniques, which we expect will further enhance its capability for precise and high-quality appearance transfer.

Acknowledgments

This work was supported by the Digital Innovation Hub project supervised by the Daegu Digital Innovation Promotion Agency (DIP), grant funded by the Korea government (MSIT and Daegu Metropolitan City) in 2025 (No. 25DIH-11, Development of Model Context Protocol (MCP)-Based Multi-Agent Collaborative System for Large Language Models (LLMs)). ※ MSIT: Ministry of Science and ICT.