Describe, Don't Dictate: Semantic Image Editing with Natural Language Intent

TL;DR

DescriptiveEdit is a novel semantic image editing framework that leverages reference images and prompts to improve editing accuracy and scalability without architectural changes or inversion, addressing limitations of previous methods.

Contribution

It re-frames instruction-based editing as reference-image-based text-to-image generation using a Cross-Attentive UNet, enhancing flexibility and performance.

Findings

Improves editing accuracy and consistency on Emu Edit benchmark.

Overcomes dataset quality limitations inherent in instruction-based models.

Seamlessly integrates with existing extensions like ControlNet and IP-Adapter.

Abstract

Despite the progress in text-to-image generation, semantic image editing remains a challenge. Inversion-based algorithms unavoidably introduce reconstruction errors, while instruction-based models mainly suffer from limited dataset quality and scale. To address these problems, we propose a descriptive-prompt-based editing framework, named DescriptiveEdit. The core idea is to re-frame `instruction-based image editing' as `reference-image-based text-to-image generation', which preserves the generative power of well-trained Text-to-Image models without architectural modifications or inversion. Specifically, taking the reference image and a prompt as input, we introduce a Cross-Attentive UNet, which newly adds attention bridges to inject reference image features into the prompt-to-edit-image generation process. Owing to its text-to-image nature, DescriptiveEdit overcomes limitations in instruction dataset quality, integrates seamlessly with ControlNet, IP-Adapter, and other extensions, and is more scalable. Experiments on the Emu Edit benchmark show it improves editing accuracy and consistency.

Tables

Table 1. Table 1 : Quantitative Comparison on the EMU-Edit test-set. We evaluate models in terms of instruction adherence (CLIP-T), image consistency (L1, L2, SSIM, DINO-I, CLIP-I, LPIPS), and image quality (PSNR). ∗ \ast indicates results from the original paper due to unavailable code. † \dagger indicates results obtained by retraining models on our training dataset. The best and second-best results are in bold and underlined .

Model	Training-based?	L1 $\downarrow$	L2 $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	DINO-I $\uparrow$	CLIP-I $\uparrow$	CLIP-T $\uparrow$
MasaCtrl [3]	✗	0.072	0.014	0.174	19.31	0.654	0.797	0.863	0.299
RF-Edit [45]	✗	0.096	0.022	0.317	17.10	0.554	0.553	0.757	0.319
PnPInversion [20]	✗	0.084	0.014	0.205	18.92	0.639	0.749	0.802	0.307
FPE [26]	✗	0.071	0.012	0.185	20.41	0.664	0.714	0.822	0.293
TurboEdit [8]	✗	0.095	0.023	0.234	17.56	0.598	0.659	0.787	0.313
IP2P† [2]	✓	0.083	0.015	0.210	20.03	0.619	0.740	0.805	0.293
MagicBrush [52]	✓	0.082	0.026	0.179	19.48	0.655	0.752	0.837	0.305
EmuEdit* [40]	✓	0.094	-	-	-	-	0.819	0.859	0.231
AnyEdit† [51]	✓	0.067	0.020	0.147	19.81	0.657	0.809	0.832	0.271
BrushEdit [24]	✓	0.143	0.050	0.272	13.51	0.451	0.601	0.782	0.299
DescribeEdit(Ours)	✓	0.065	0.011	0.139	20.99	0.661	0.843	0.874	0.315

Table 2. Table 2 : Quantitative comparison of description-based and instruction-based inputs for fine-tuning T2I models. The description-based approach achieves higher performance, demonstrating better alignment with the pre-trained T2I priors.

Method	CLIP-T $\uparrow$	DINO-I $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$
Description	0.284	0.741	0.562	18.309
Instruction	0.272	0.739	0.551	18.123

Table 3. Table 3 : Ablation study on the attention fusion strategies.

Method	CLIP-T $\uparrow$	DINO-I $\uparrow$	SSIM $\uparrow$	PSNR $\uparrow$
Direct Replacement	0.3005	0.6690	0.4261	13.78
Direct Addition	0.3052	0.7532	0.4970	14.77
Ours	0.3162	0.7931	0.6153	18.58

Equations

$${\mathbf{I}}_e\leftarrow {\mathcal{F}}_{\mathbf{\theta }}({\mathbf{T}}_e|{\mathbf{I}}_o),$$

$${\mathbf{Z}}^{\prime }=\text{CA}(\mathbf{Q},{\mathbf{K}}_{\mathbf{c}},{\mathbf{V}}_{\mathbf{c}})=\text{softmax}\left(\frac{\mathbf{Q}{\mathbf{K}}_{\mathbf{c}}^{\mathsf{T}}}{\sqrt{d}}\right)\cdot {\mathbf{V}}_{\mathbf{c}},$$

$$\mathbf{Q}=\mathbf{Z}{\mathbf{W}}_q,{\mathbf{K}}_{\mathbf{c}}={\mathbf{f}}_{\mathbf{c}}{\mathbf{W}}_k,{\mathbf{V}}_{\mathbf{c}}={\mathbf{f}}_{\mathbf{c}}{\mathbf{W}}_v.$$

$${\mathbf{Z}}^{\prime }=\text{CA}({\mathbf{Q}}_{{\mathbf{I}}_o},{\mathbf{K}}_{{\mathbf{T}}_e},{\mathbf{V}}_{{\mathbf{T}}_e}).$$

$${\mathbf{Z}}^{\text{in}}=\mathbf{Z}+\text{Linear}({\mathbf{Z}}^{\prime }),$$

$$\mathcal{L}={\mathbb{E}}_{{\mathbf{Z}}_e^0,{\mathbf{Z}}_o^0,\mathbf{ϵ},t,s}\left[\Vert \mathbf{ϵ}-{\mathbf{ϵ}}_{\theta }({\mathbf{Z}}_e^t,t,{\mathbf{T}}_e,{\mathbf{Z}}_o^s,s){\Vert }^2\right],$$

$${\widetilde{\mathbf{ϵ}}}_{\theta }({\mathbf{Z}}_e^t,{\mathbf{Z}}_o^s,{\mathbf{T}}_e)$$

$$\displaystyle+\lambda_{I}\cdot\big{(}\boldsymbol{\epsilon}_{\theta}(\boldsymbol{Z}_{e}^{t},\boldsymbol{Z}_{o}^{s},\varnothing)-\boldsymbol{\epsilon}_{\theta}(\boldsymbol{Z}_{e}^{t},\varnothing,\varnothing)\big{)}$$

$$\displaystyle+\lambda_{T}\cdot\big{(}\boldsymbol{\epsilon}_{\theta}(\boldsymbol{Z}_{e}^{t},\boldsymbol{Z}_{o}^{s},\boldsymbol{T}_{e})-\boldsymbol{\epsilon}_{\theta}(\boldsymbol{Z}_{e}^{t},\boldsymbol{Z}_{o}^{s},\varnothing)\big{)},$$

Full text

††$\star$ Equal contributions. \faEnvelopeCorrespondence to: Ying Tai.

1 Introduction

Text-to-image (T2I) generation—synthesizing images from textual prompts describing desired content—has made significant breakthroughs [22, 9, 6, 32], driven by advances in three key areas: large high-quality datasets (e.g., LAION-5B [39]), efficient and scalable network architectures (e.g., Latent Diffusion Models [36], Diffusion Transformers [31]), and theoretical progress in modeling high-dimensional data distributions [25, 23, 16, 43]. However, semantic image editing—“generating” what people want to change on the given images (still is text-guided)—remains fundamentally challenging.

Inversion-based techniques [14, 44, 42, 38, 11, 45, 56] invert the input image to a noisy latent and regenerate a new image guided by an edit prompt (e.g., “a smiling person”). However, the inverting process suffers from reconstruction errors and inefficiency due to its iterative optimization. Recently, modifying the T2I model architecture and training it to adhere to specific editing instructions (e.g., “make the person smile”) yields harmonious and accurate editing results [19, 55, 12, 51, 2]. However, there are two limitations we need to consider. First is the data bottleneck. Collecting a large-scale dataset of instructional training data—comprised of input images, edited images, and edit instructions—is both challenging and costly. Compared to popular T2I generation dataset (e.g., LAION-5B), existing instruction-based datasets, such as AnyEdit [51] ($\sim$2.5M) and UltraEdit [55] ($\sim$4M) are significantly smaller and less diverse. Second, modifying the architecture entails substantial computational costs for retraining the diffusion model and can hinder compatibility with diverse pretrained T2I models and their associated ecosystem tools (e.g., ControlNet [53], IP-Adapter [50]).

These challenges raise a key question: Can we “generate” the desired editing intent directly from descriptive prompts instead of relying on instructions? If so, text-to-image generation and editing would merge into a unified computational framework, inherently solving the issues of training data scale and compatibility. Moreover, the descriptive prompt contains more details, enabling more precise edits. The underlying motivation is that instruction-based editing can be equivalently understood as a two-stage process: “instructions $\rightarrow$ edit descriptions $\rightarrow$ edit images”.

To answer this question, we propose DescriptiveEdit, a novel image editing framework that reinterprets the role of text guidance by transitioning from instruction-based to description-driven editing. Instead of modifying diffusion model architectures or requiring additional training on instruction-labeled datasets, DescriptiveEdit leverages pre-trained T2I models in a plug-and-play manner, preserving their original generative capacity. To achieve this, we introduce a Cross-Attentive UNet, which injects reference image features into the denoising process via a lightweight attention bridge (newly added attention layers). This mechanism enables the model to align the edited output with the structural and semantic cues of the original image, while ensuring faithful adherence to the descriptive prompt. Moreover, instead of modifying the model’s core architecture, we apply low-rank parameter tuning (LoRA) [17], enabling parameter-efficient adaptation without disrupting the pre-trained generative capabilities.

Furthermore, DescriptiveEdit seamlessly integrates with existing T2I pipelines without requiring model retraining. Unlike instruction-based methods such as IP2P [2], which fine-tune the full UNet ($\sim$860M parameters), our approach introduces only a lightweight set of trainable parameters ($\sim$75M) while maintaining superior editing accuracy and structural consistency. Additionally, DescriptiveEdit remains fully compatible with community-driven extensions, such as ControlNet [53] and IP-Adapter [50], allowing enhanced control over generation while preserving the flexibility of pre-trained models.

In summary, our contributions are as follows:

• •

We re-examine the role of T2I models in image editing and propose a description-based semantic editing paradigm that preserves generative quality.

• •

We present a non-invasive adaptation framework that merges attention-based feature projection with parameter-efficient tuning, providing both flexibility and reusability.

• •

Extensive experiments demonstrate our method’s superior performance in editing accuracy and image consistency compared to existing approaches.

2 Related Work

Text-to-Image Generation.

Modern text-to-image synthesis [13, 34, 29, 35, 37, 10, 22] has undergone three evolutionary phases. Early GAN-based approaches [13] struggled with mode collapse and limited diversity. Inspired by nonequilibrium thermodynamics [41], the diffusion revolution began with GLIDE [29], which established pixel-space text-image alignment through iterative denoising. Latent diffusion models (LDM) [37] later improved efficiency by operating in compressed feature space while maintaining fidelity. Recent architectural innovations like Diffusion Transformers (DiT) [30] replace convolutional UNets with pure attention mechanisms, enabling superior global coherence and scalability. Parallel developments in flow-based models [22] offer alternative probabilistic mappings from noise to image space. These advancements collectively establish the foundation for semantic image editing.

Semantic Image Editing.

Semantic image editing demands precise modifications while preserving fidelity, making it more challenging than T2I generation. Existing methods fall into training-based and training-free approaches. Training-based methods improve control but often degrade generative quality. Some fine-tune entire models [47, 2, 52, 55], causing artifacts, while others use MLLMs[12, 19] or MoE architectures[40, 51, 49], which enhance task-specific performance but require costly retraining. Training-free methods exploit pre-trained models but struggle with efficiency and accuracy. Early works modified noise levels[1], later methods adjusted attention mechanisms[14, 44, 3, 27], and Flux-based models [45, 38, 11, 7] improved sampling efficiency. However, inversion reliance increases computational cost. We introduces an efficient training paradigm that preserves generative ability while enabling general-purpose editing via descriptive text, achieving a better trade-off between control, efficiency, and generalization.

3 DescriptiveEdit

In this work, we propose, DescriptiveEdit, a novel approach to text-guided semantic image editing by describing the final desired effect rather than providing direct editing instructions as in conventional instruction-based methods. In particular, DescriptiveEdit leverages pre-trained T2I models as a fixed foundational component without altering their weights, thereby avoiding any modifications to the architecture or additional training processes.

The remainder of the paper is organized as follows. In Sec. 3.1, we formally define the problem. Sec. 3.2 details the framework and its key components. Finally, Sec. 3.3 presents the procedures for training and inference.

3.1 Problem Formulation

Instruction-based image editing works formulate the semantic image editing task as follows: given an original image and the instructions describing the desired modifications (e.g. add firework to the sky), a model is trained to generate the corresponding edited output. In our approach, we reframe this process into two distinct stages: “instructions $\rightarrow$ edit descriptions $\rightarrow$ edited images.” Here, the edit description (e.g., fireworks are blooming in the sky) can be either user-specified or automatically derived from the original image and instructions using large visual language models (e.g., QWen-VL [46]). The unexplored challenge, therefore, is to design a model that can effectively translate an edit description into the corresponding edited image.

Formally, given the edit description $\boldsymbol{T}_{e}$ and the original image $\boldsymbol{I}_{o}$, the formulation of our DescriptiveEdit is:

[TABLE]

where $\boldsymbol{I}_{e}$ is the edited image, $\mathcal{F}_{\boldsymbol{\theta}}$ represents the editing model, and $\boldsymbol{\theta}$ is its parameters. $\boldsymbol{T}_{e}$ actually is the “prompt” in context of T2I models, and thus $\mathcal{F}_{\boldsymbol{\theta}}$ is essentially a T2I diffusion model conditioned on the original image $\boldsymbol{I}_{o}$.

But, since the T2I model does not support image-based conditioning, we seek adjustments. Unlike InstructPix2Pix, which modifies structure and input at the cost of compatibility and training efficiency, we aim for a lightweight, flexible solution that conditions on the original image without altering the model’s architecture or pretrained weights.

3.2 Framework Architecture

As illustrated in Fig. 2, we introduce an attention fusion mechanism that operates within the latent space of the frozen model, enabling effective integration of original image $\boldsymbol{I}_{o}$ information into the editing process (i.e., the edit description $\boldsymbol{T}_{e}$ $\rightarrow$ the edited image $\boldsymbol{I}_{e}$). To ensure efficiency, we adopt low-rank parameter updates using LoRA [17].

3.2.1 Cross-Attentive UNet with Attention Bridge

LDMs commonly employ two conditioning schemes: cross-attention (e.g., Stable Diffusion [37], IP-Adaptor [50]) and a separate UNet for feature extraction (e.g., ControlNet, Animate-Anyone [18]), each with its own strengths and limitations. Cross-attention (CA) is a lightweight conditioning mechanism with minimal training overhead. It integrates conditional information $\boldsymbol{c}$ via the general formulation:

[TABLE]

where

[TABLE]

Here, $\boldsymbol{Z}$ is the intermediate representation of the UNet, $\boldsymbol{f}_{\boldsymbol{c}}$ represents the feature encoding of the conditioning input $\boldsymbol{c}$, $\boldsymbol{W}_{q}$, $\boldsymbol{W}_{k}$, and $\boldsymbol{W}_{v}$ are learnable projection matrices. Although CA-based conditioning is computationally efficient, the input feature $\boldsymbol{f}_{\boldsymbol{c}}$ primarily encodes high-level semantics, enforcing only semantic relevance to the condition $\boldsymbol{c}$ (as also discussed in Animate-Anyone). For low-level alignment, an alternative approach involves using a separate UNet (a copy of the denoising UNet) to extract features from the conditional image. These features are then fused with the denoising UNet via channel-wise concatenation (refer to ControlNet). However, this modification alters the input feature dimensions, requiring fine-tuning on large-scale datasets, which significantly increases training costs.

To seamlessly integrate the strengths of both conditioning schemes—cross-attention and separate UNet encoding—we propose a novel image-conditioning algorithm for semantic image editing. To enable low-level alignment the edit output and the input image, we still use a separate UNet (so-called Ref-UNet, inherent weights from the denoising UNet) to encode the input images. At the self-attention layers of both UNets 111It’s a common practice since self-attention dominates spatial information [14, 29, 21]., we perform cross-attention by taking the key ($\boldsymbol{K}_{\boldsymbol{T}_{e}}$) and value ($\boldsymbol{V}_{\boldsymbol{T}_{e}}$) from the denoising UNet’s self-attention and the query ($\boldsymbol{Q}_{\boldsymbol{I}_{o}}$) from the Ref-UNet’s self-attention (see Eq. 2). The formulation is:

[TABLE]

$\boldsymbol{Z}^{\prime}$ is then added to $\boldsymbol{Z}$ (the self-attention output of the denoising UNet ). The newly introduced cross-attention mechanism acts as a bridge, enabling the denoising UNet to incorporate information from the Ref-UNet, effectively forming a Cross-Attentive UNet.

Our proposed cross-attentive denoising scheme requires training only the newly added CA layers, while all other parameters remain frozen. Moreover, it is important to note that cross-attentive denoising is not restricted to UNet-based architectures; it is also compatible with DiT-based architectures. By replacing UNet with DiT and applying the cross-attention mechanism used in DiT, the same framework can be extended. Refer to Fig. 7 for qualitative results.

3.2.2 Balancing Reference and Editing

A straightforward approach to incorporating reference features is to directly add the self-attention output of the denoising UNet, $\boldsymbol{Z}$, with the reference-enhanced feature, $\boldsymbol{Z}^{\prime}$. However, this manner often overpowers the original features and nullifies the intended editing effect. Further analysis is provided in Appendix.

To address this, we add a learnable linear mapping to establish a dynamic equilibrium between these components:

[TABLE]

where $\text{Linear}(\cdot)$ is a learnable transformation initialized with zero weights. This design offers two key advantages: (1) it adaptively balances the generative prior ($\boldsymbol{Z}$) and reference guidance ($\boldsymbol{Z}^{\prime}$) by learning optimal contribution weights; (2) zero initialization ensures $\text{Linear}(\boldsymbol{Z}^{\prime})\approx 0$ at the start of training, preserving the base model’s behavior while gradually integrating reference features.

3.3 Training and Inference

Training.

Inspired by classifier-free guidance [15], we randomly set edit descriptions and original images empty for 5% of examples independently during training. Both the edited image $\boldsymbol{Z}_{e}$ and the original image $\boldsymbol{Z}_{o}$ are simultaneously fed into the model. But, if the same noise is applied to them, crucial reference information may be lost, hindering the model’s ability to learn the editing capability. Inspired by Diffusion Forcing [5], where the model is trained with inputs containing independent noise levels, we only apply noise to the edited image. For parameter-efficient adaptation, we initialize the attention bridge layers by copying the weights from pretrained self-attention layers and fine-tune them using LoRA [17]. Meanwhile, the newly introduced learnable mapping layers follow ControlNet [53]’s zero-initialization strategy to preserve the original model behavior at initialization. Finally, we minimize the following latent diffusion objective:

[TABLE]

where $t$ is the diffusion timestep for the edited image, $s$ is the fixed timestep for the original image set to $s=0$, $\boldsymbol{Z}_{e}^{t}$ denotes the noisy edited image latent representation at timestep $t$ and $\boldsymbol{Z}_{o}^{s}$ represents the clean original image latent representation at timestep $s$.

Inference.

Since our training process incorporates conditional information from the original image in addition to the edited image and edit description, we redefine the denoising formulation, drawing inspiration from IP2P [2]. The modified denoising process is formulated as follows:

[TABLE]

The terms $\lambda_{I}$ and $\lambda_{T}$ control the guidance strength for the original image and textual conditions, respectively. This formulation ensures that the model effectively integrates both textual and original image conditions while maintaining its generative capabilities.

4 Experiments

4.1 Experimental Setup

Training Details.

To ensure fair comparisons, we adopt Stable Diffusion v1.5 [37] as the backbone, aligning with most training-based methods [2, 52, 51]. The model is optimized using the AdamW [28] optimizer with a learning rate of $1\times 10^{-5}$. For parameter-efficient fine-tuning, we apply LoRA [17] to the attention bridge layers, setting the rank to 64 and $\alpha$ to 64. Additional training protocols are detailed in Appendix A.

Datasets.

Our model is trained on the UltraEdit dataset [55], which contains approximately 4M text-image pairs. For evaluation, we use the Emu Edit Test benchmark [40] following [40, 51, 55]. We found inconsistencies in this benchmark’s quality, where identical captions (e.g., “a train station in a city”) are assigned to both source and target images, regardless of their actual visual content. To ensure a fair and sound evaluation, we manually filter out non-compliant samples before calculating the metrics.

Baselines.

We compare our approach against representative baselines from two categories: (1) Training-free methods, including MasaCtrl [3], RF-Edit [45],PnPInversion [20], FPE [26], and Turboedit [8]. (2) Training-based approaches, including InstructPix2Pix [2], MagicBrush [52], EmuEdit [40], AnyEdit [51], and BrushEdit [24]. Comprehensive baseline specifications are provided in Appendix B.

Evaluation Metrics.

We evaluate the performance of semantic image editing models from three perspectives: (1) Instruction Adherence: We use the CLIP-T score [33] to quantify the alignment between the edit description and the edited image. (2) Image Consistency: Pixel-level preservation is measured using L1 and L2 distances between original and edited images. Semantic-level consistency is assessed using CLIP-I [33] and DINO-I [4] feature similarities, while structural integrity is evaluated via the Structural Similarity Index (SSIM) [48]. Additionally, we adopt the Learned Perceptual Image Patch Similarity (LPIPS) [54] to evaluate perceptual consistency. (3) Image Quality: We employ the Peak Signal-to-Noise Ratio (PSNR) to assess pixel-wise reconstruction accuracy. Comprehensive metric implementations are provided in Appendix C.

4.2 Quantitative Comparison

The quantitative results are reported in Tab. 1. We compared our method with both training-free and training-based baselines across multiple evaluation metrics. In terms of image consistency, our method exhibits the lowest L1 (0.065) and L2 (0.011) distances, ensuring minimal pixel-level deviation from the reference. Moreover, it achieves the highest DINO-I (0.843) and CLIP-I (0.874) scores, demonstrating superior semantic and feature-level consistency. Additionally, our model obtains the best LPIPS score (0.139) and a competitive SSIM score (0.661), highlighting its strong perceptual similarity and structural coherence with the reference image. The slightly higher SSIM of FPE likely stems from its focus on preserving spatial structure, which reduces flexibility for non-rigid edits. Regarding instruction adherence, our model achieves a competitive CLIP-T score (0.315), which is comparable to RF-Edit (0.319), indicating strong alignment between generated images and textual instructions. For image quality, our model outperforms all baselines with the highest PSNR (20.99), indicating superior pixel-wise reconstruction accuracy. The slightly lower CLIP-T score compared to RF-Edit likely results from the inherent characteristics of training-free methods, which prioritize instruction adherence at the expense of fine-grained visual consistency. Overall, our results suggest that training-based methods like ours can achieve a more optimal trade-off, ensuring high-fidelity, description-driven edits while maintaining strong consistency with reference images. This highlights the practical utility of our approach in real-world semantic image editing tasks.

4.3 Qualitative Comparison

Fig. 4 shows the qualitative comparisons on global and local editing tasks. Our method outperforms both training-based and training-free approaches, including the flux-based RF-Edit, by achieving high-fidelity edits with strong semantic consistency, while existing methods lack robustness, often failing drastically in certain cases.

For global editing, our model effectively transforms styles and replaces backgrounds while preserving key foreground elements. As shown in Fig. 4, it successfully integrates new backgrounds, such as volcanoes or snow-covered mountains, without introducing artifacts—whereas existing methods often fail to modify the background correctly or disrupt structural integrity. In local editing, our approach seamlessly modifies objects—e.g., replacing bread with a mice or adding glasses to a cat—while maintaining coherence, a challenge for prior methods.

4.4 Ablations

Description vs. Instruction.

As discussed in Sec.1, T2I models are pre-trained on large-scale description-image data, making descriptions more compatible than instructions for image editing. We conduct a experiment by varying only the text input type and observe that the description-based method consistently outperforms the instruction-based one across multiple metrics (Tab.2). Beyond quantitative results, Fig. 3 illustrates that description prompts lead to more accurate edits under the same intentions. This demonstrates that free-form descriptions offer better guidance by providing richer details and clearer intent, resulting in improved editing fidelity.

Effect of $\lambda_{I}$ on Image Editing.

To analyze the impact of $\lambda_{I}$, we conduct an ablation study by varying its intensity and observing the model’s behavior. The results are visualized in Fig. 5, illustrating seasonal transitions. When $\lambda_{I}$ is small, the model exhibits strong editing capabilities, leading to more pronounced seasonal effects, such as denser snowfall and more vibrant foliage. However, this comes at the cost of lower similarity to the reference image. Conversely, with a larger $\lambda_{I}$, the model prioritizes reference image fidelity, reducing its ability to perform significant edits. Through empirical observation, we find that setting $\lambda_{I}$ in the range of 1 to 2.5 achieves a favorable balance between semantic editing strength and reference image preservation. This allows users to adjust $\lambda_{I}$ based on specific needs, trading off between edit intensity and reference consistency.

Adaptive Attention Fusion

To verify the effectiveness of our adaptive attention fusion in Sec. 3.2.2, we conduct an ablation study comparing alternative fusion strategies:

• •

Direct Replacement:$\boldsymbol{Z}^{\text{in}}=\boldsymbol{Z}^{\prime}$, assuming full reliance on the reference-enhanced feature map.

• •

Direct Addition: $\boldsymbol{Z}^{\text{in}}=\boldsymbol{Z}+\boldsymbol{Z}^{\prime}$, treating both components as equally important.

• •

Ours: $\boldsymbol{Z}^{\text{in}}=\boldsymbol{Z}+\text{Linear}(\boldsymbol{Z}^{\prime})$, where a learnable transformation dynamically adjusts the integration.

The results are reported in Tab. 3. Direct replacement performs the worst across all metrics, demonstrating that sole reliance on $\boldsymbol{Z}^{\prime}$ severely degrades image fidelity and structure preservation. Direct addition mitigates some of these issues but still lacks adaptive control, leading to suboptimal balance. Our approach, leveraging a learnable linear transformation, achieves the best equilibrium by dynamically adjusting reference influence while preserving generative capability. This validates the necessity of adaptive feature integration for high-quality image editing.

4.5 Compatibility with Community Extensions

A key advantage of our approach is its seamless compatibility with community-driven extensions, enabling flexible integration without modifying the base model or requiring retraining. Unlike prior methods that impose architectural constraints or demand fine-tuning, our design preserves the diffusion model’s structure while naturally incorporating external control signals.

We validate this capability with three representative models: IP-Adapter[50], ControlNet[53], and RealCartoon3D, each serving distinct purposes—IP-Adapter and ControlNet enhance conditional control through structural and style guidance, while RealCartoon3D specializes in artistic stylization. As shown in Fig. 6, our method seamlessly integrates with each: IP-Adapter and ControlNet transform scenes into winter settings with realism and structural consistency, while RealCartoon3D produces high-quality cartoon-style conversions without artifacts.

These results underscore our model’s plug-and-play adaptability, extending diffusion-based editors without altering their core architecture. By ensuring compatibility across diverse editing paradigms, our approach enhances the model’s versatility and practical applicability.

4.6 Robustness Across Architectures

To demonstrate the adaptability of our approach across different diffusion architectures, we apply it to Flux [22], showcasing its seamless integration with DiT-based model. As shown in Fig. 7, our approach consistently produces high-fidelity edits while preserving structural integrity, confirming the robustness of our adaptive feature integration strategy. By eliminating architecture-dependent constraints, our method ensures reliable performance across diverse diffusion-based frameworks, reinforcing its practicality for real-world applications.

5 Conclusions

We proposed a description-based method, DescriptiveEdit, that unifies semantic image editing to the text-to-image generation framework. An attention bridge enables efficient feature integration, while LoRA-based tuning ensures parameter efficiency and compatibility. Experiments show superior editing accuracy and consistency, establishing a scalable, plug-and-play paradigm for semantic image editing.

Acknowledgements

This work was supported by Natural Science Foundation of China: No. 62406135, Natural Science Foundation of Jiangsu Province: BK20241198, and the Gusu Innovation and Entrepreneur Leading Talents: No. ZXL2024362. This work was supported in part by the Shanghai Municipal Commission of Economy and Informatization, under Grant No. 2024-GZL-RGZN-01008.