Audio-Guided Visual Editing with Complex Multi-Modal Prompts

TL;DR

This paper presents an audio-guided visual editing framework that effectively handles complex multi-modal editing tasks without additional training, leveraging a pre-trained encoder and innovative prompt integration techniques.

Contribution

It introduces a novel multi-modal editing approach using a pre-trained encoder and separate noise branching to manage complex scenarios with multiple prompts.

Findings

Outperforms existing methods in complex editing scenarios

Handles multiple and multi-modal prompts effectively

No additional training required for new tasks

Abstract

Visual editing with diffusion models has made significant progress but often struggles with complex scenarios that textual guidance alone could not adequately describe, highlighting the need for additional non-text editing prompts. In this work, we introduce a novel audio-guided visual editing framework that can handle complex editing tasks with multiple text and audio prompts without requiring additional training. Existing audio-guided visual editing methods often necessitate training on specific datasets to align audio with text, limiting their generalization to real-world situations. We leverage a pre-trained multi-modal encoder with strong zero-shot capabilities and integrate diverse audio into visual editing tasks, by alleviating the discrepancy between the audio encoder space and the diffusion model's prompt encoder space. Additionally, we propose a novel approach to handle complex scenarios with multiple and multi-modal editing prompts through our separate noise branching and adaptive patch selection. Our comprehensive experiments on diverse editing tasks demonstrate that our framework excels in handling complicated editing scenarios by incorporating rich information from audio, where text-only approaches fail.

Tables

Table 1. Table 1 : Evaluation results for A2I-edit and TA2I-edit . Left: quantitative scores; right: user study. Str. Dist is structure distance. CLIP_Audio , CLIP_Text , and CLIP_integrated are CLIP scores w.r.t. audio, text, and integrated captions. User ratings—Structural Pres., Audio Align., Text Align., Overall—compare ours with the baseline; bold indicates the best result.

		Str. Dist$(\downarrow )$	LPIPS$(\downarrow )$	CLIP_Audio$(\uparrow )$	CLIP_Text$(\uparrow )$	CLIP_integrated$(\uparrow )$	Structural Preservation$(\uparrow )$			Audio Alignment$(\uparrow )$			Text Alignment$(\uparrow )$			Overall Alignment$(\uparrow )$
							Win	Tie	Lose	Win	Tie	Lose	Win	Tie	Lose	Win	Tie	Lose
A2I-edit	AVStyle	0.0396	0.2219	14.5596	-	-	88.40	7.60	4.00	90.06	2.59	7.34	-	-	-	-	-	-
	Sonic Diffusion	0.0475	0.2915	13.4908	-	-	77.38	4.96	17.66	91.27	4.15	4.59	-	-	-	-	-	-
	P2P (caption)	0.0617	0.3261	12.2536	-	-	89.66	2.87	7.47	82.31	11.13	6.56	-	-	-	-	-	-
	NT (caption)	0.0097	0.1145	12.7248	-	-	78.85	10.04	11.11	86.41	8.70	4.88	-	-	-	-	-	-
	PnP (caption)	0.0227	0.1866	12.8969	-	-	74.02	3.52	22.46	75.04	8.56	16.40	-	-	-	-	-	-
	Ours	0.0341	0.2539	17.0472	-	-	-	-	-	-	-	-	-	-	-	-	-	-
TA2I-edit	AVStyle $\to$ NT	0.0668	0.4174	13.1312	25.2388	21.1613	68.69	5.05	26.26	70.42	5.52	24.06	58.92	11.42	29.66	70.88	13.65	15.48
	NT $\to$ AVStyle	0.0369	0.2508	14.4867	25.5348	21.6772	56.59	14.10	29.31	71.73	20.90	7.36	46.12	14.88	38.99	59.15	17.41	23.44
	P2P (caption)	0.0591	0.3278	12.3059	24.9569	21.1932	56.47	8.04	35.49	69.48	11.69	18.83	48.64	14.40	36.96	76.60	7.35	16.05
	NT (caption)	0.0192	0.1802	12.8379	24.3057	21.6057	56.59	14.10	29.31	71.73	20.90	7.36	46.12	14.88	38.99	59.15	17.41	23.44
	PnP (caption)	0.0264	0.2108	13.0695	24.9375	21.3103	50.83	16.39	32.78	76.88	7.89	15.23	48.99	17.31	33.70	48.12	27.33	24.55
	Sonic Diffusion	0.0179	0.1551	12.9011	23.6654	20.1574	49.80	9.92	40.28	63.12	14.17	22.71	51.22	17.64	31.14	69.61	12.53	17.87
	Ours (mean)	0.0204	0.1729	14.0025	24.2257	20.1574	39.69	18.51	41.79	55.33	22.79	21.88	27.36	48.69	23.94	39.71	39.92	20.37
	Ours	0.0333	0.2392	15.0699	24.1283	23.0356	-	-	-	-	-	-	-	-	-	-	-	-

Table 2. Table 2 : Evaluation results for Audio-guided Video Editing (A2V-edit) and combined Text and Audio-guided Video Editing (TA2V-edit). CLIP_frame represents the frame-wise CLIP similarity score. All other abbreviations are consistent with those used in Table 1 .

		CLIP_frame$(\uparrow )$	CLIP_Audio$(\uparrow )$	CLIP_Text$(\uparrow )$	CLIP_integrated$(\uparrow )$	Structural Preservation$(\uparrow )$			Audio Alignment$(\uparrow )$			Text Alignment$(\uparrow )$			Overall Alignment$(\uparrow )$
						Win	Tie	Lose	Win	Tie	Lose	Win	Tie	Lose	Win	Tie	Lose
A2V-edit	Tokenflow (caption)	0.9060	15.2095	-	-	61.98	8.89	29.14	61.25	10.47	28.29	-	-	-	-	-	-
	Gen1 (caption)	0.8854	14.5752	-	-	57.07	2.31	40.62	70.77	6.52	22.71	-	-	-	-	-	-
	Ours	0.9029	18.1445	-	-	-	-	-	-	-	-	-	-	-	-	-	-
TA2V-edit	Tokenflow (caption)	0.9162	13.2351	23.6890	22.4227	43.50	19.28	37.22	54.84	9.30	35.86	49.78	21.34	28.88	58.40	9.02	32.58
	Gen1 (caption)	0.8780	14.8816	23.7246	22.8530	54.46	0.48	45.06	48.41	10.72	40.87	50.39	5.17	44.44	49.79	8.58	41.63
	Ours (mean)	0.9082	15.4571	22.9909	22.7029	48.67	10.40	40.93	50.54	37.91	11.55	35.46	39.38	25.15	38.19	38.56	23.25
	Ours	0.9043	16.1833	22.8336	23.2885	-	-	-	-	-	-	-	-	-	-	-	-

Equations

$$\mathcal{L}_{ldm}=\mathbb{E}_{\mathcal{E}(x),\epsilon\sim\mathcal{N}(0,1),t}\lVert\epsilon-\epsilon_{\theta}(\mathbf{z}_{t},t,\mathcal{C})\lVert_{\scalebox{0.7}{$2$}}^{\scalebox{0.7}{$2$}},$$

$${\mathbf{z}}_{t+1}^{\ast }=\sqrt{\frac{{\alpha }_{t+1}}{{\alpha }_t}}{\mathbf{z}}_t^{\ast }+\left(\sqrt{\frac{1}{{\alpha }_{t+1}}-1}-\sqrt{\frac{1}{{\alpha }_t}-1}\right)\cdot {ϵ}_{\theta }({\mathbf{z}}_t^{\ast },t,{\mathcal{C}}_{\text{inv}})$$

$$\mathbf{c}_{\mathrm{CLIP}}=\frac{\mathbf{M}\,\mathbf{c}_{\mathrm{SD,pooled}}}{\bigl{\lVert}\mathbf{M}\,\mathbf{c}_{\mathrm{SD,pooled}}\bigr{\rVert}},$$

$$\tilde{\mathbf{c}}_{\mathrm{SD}}=\bigl{(}\mathbf{M}^{\!\top}\mathbf{M}+\lambda I\bigr{)}^{-1}\,\mathbf{M}^{\!\top}\Bigl{(}\lVert\mathbf{c}_{\mathrm{inv}}\rVert\,\mathbf{c}_{A}\Bigr{)},$$

$${ϵ}_i={ϵ}_{\theta }({\mathbf{z}}_t,t,{\mathcal{C}}_i),i=1,\dots ,N.$$

$${\widehat{ϵ}}_{\theta }(p)={ϵ}_{\text{inv}}(p)+\Delta {ϵ}_{i^{\star }(p)}(p),i^{\star }(p)=\arg \underset{i}{\max }\Vert \Delta {ϵ}_i(p)\Vert \text{vskip}-10.0pt$$

Full text

1 Introduction

Recent advancements in text-to-image diffusion models have facilitated visual editing tasks, which require precise manipulation of images or videos while preserving essential elements [Hertz et al.(2022)Hertz, Mokady, Tenenbaum, Aberman, Pritch, and Cohen-Or, Mokady et al.(2023)Mokady, Hertz, Aberman, Pritch, and Cohen-Or, Tumanyan et al.(2023)Tumanyan, Geyer, Bagon, and Dekel, Geyer et al.(2023)Geyer, Bar-Tal, Bagon, and Dekel]. However, relying solely on text prompts limits their effectiveness in complex editing scenarios where descriptions may be ambiguous or insufficient [Kumari et al.(2023)Kumari, Zhang, Zhang, Shechtman, and Zhu, Brack et al.(2024)Brack, Friedrich, Kornmeier, Tsaban, Schramowski, Kersting, and Passos, Zhao et al.(2024)Zhao, Chen, Chen, Bao, Hao, Yuan, and Wong, He et al.(2024)He, Zheng, Fang, Piramuthu, Bansal, Ordonez, Sigurdsson, Peng, and Wang]. To address this limitation, researchers have incorporated non-text conditions—such as edge map, segmentation mask, and depth map—to provide richer and more precise control over the editing process [Mo et al.(2024)Mo, Mu, Lin, Liu, Guan, Li, and Zhou, Zhao et al.(2024)Zhao, Chen, Chen, Bao, Hao, Yuan, and Wong, He et al.(2024)He, Zheng, Fang, Piramuthu, Bansal, Ordonez, Sigurdsson, Peng, and Wang, Lin et al.(2024)Lin, Cho, Zala, and Bansal]. Audio is one of the widely used modalities and can provide rich and dynamic information, thus being effective for visual editing. However, in audio-guided visual editing, existing methods often require additional training on specific datasets to align audio with the text [Yang et al.(2023)Yang, Zhang, Ge, Shao, Xue, Qiao, and Luo, Biner et al.(2024)Biner, Sofian, Karakaş, Ceylan, Erdem, and Erdem, Li et al.(2022)Li, Liu, Owens, and Zhao], limiting their generalization to diverse real-world scenarios.

We address this limitation by leveraging a pre-trained aligned multi-modal encoder trained on large-scale text-audio and audio-visual datasets [Tang et al.(2023b)Tang, Yang, Zhu, Zeng, and Bansal], which demonstrates strong zero-shot capabilities. However, integrating this encoder with diffusion models such as Stable Diffusion [Rombach et al.(2022)Rombach, Blattmann, Lorenz, Esser, and Ommer] faces challenges due to the discrepancy between the encoder’s output space and the CLIP text encoder space used in Stable Diffusion, as illustrated in Figure 1. We overcome this incompatibility by introducing a simple mapping technique involving only a matrix inversion, effectively incorporating audio features in visual editing without extensive retraining.

Achieving richer and more complex visual editing often requires integrating multiple editing prompts [Mo et al.(2024)Mo, Mu, Lin, Liu, Guan, Li, and Zhou, Zhao et al.(2024)Zhao, Chen, Chen, Bao, Hao, Yuan, and Wong, He et al.(2024)He, Zheng, Fang, Piramuthu, Bansal, Ordonez, Sigurdsson, Peng, and Wang, Lin et al.(2024)Lin, Cho, Zala, and Bansal], such as combining audio and text prompts. Existing approaches typically enhance diffusion models with additional modules to handle multiple editing prompts, requiring further training. In contrast, we propose a novel separate noise branching and adaptive patch selection as shown in Figure 1, which can effectively integrate multiple editing signals without training.

To comprehensively evaluate our framework, we construct new benchmark datasets, PIEBench-multi and DAVIS-multi, by extending existing text-guided visual editing benchmarks to include audio editing prompts. Through comprehensive experiments and analysis, we demonstrate that our framework effectively handles complex editing scenarios by effectively incorporating rich information from audio, which existing text-only methods struggle with. We will publicly release our benchmarks to promote subsequent research.

Our contributions can be summarized as follows:

• •

We present a novel zero-shot approach to integrate audio into visual editing, which delivers strong performance across diverse types of audio by effectively integrating a pre-trained aligned multi-modal encoder without any further training.

• •

We propose a novel approach of separate noise branching and adaptive patch selection, which can handle complex visual editing scenarios with multiple editing prompts without further training.

• •

Comprehensive experiments and analysis on our new benchmark datasets, PIEBench-multi and DAVIS-multi, demonstrate that our framework effectively handles complex editing scenarios involving both text and audio prompts, where existing text-only methods struggle. We will also release our benchmarks to promote subsequent research.

2 Related Work

2.1 Diffusion Models in Multi-modal Generation

With the groundbreaking achievements in text-to-image diffusion models [Rombach et al.(2022)Rombach, Blattmann, Lorenz, Esser, and Ommer, Ramesh et al.(2022)Ramesh, Dhariwal, Nichol, Chu, and Chen, Saharia et al.(2022)Saharia, Chan, Saxena, Li, Whang, Denton, Ghasemipour, Gontijo Lopes, Karagol Ayan, Salimans, et al., Podell et al.(2023)Podell, English, Lacey, Blattmann, Dockhorn, Müller, Penna, and Rombach], researchers have actively explored the application of diffusion models in various output modalities, including text [Gong et al.(2022)Gong, Li, Feng, Wu, and Kong, Wu et al.(2024)Wu, Fan, Liu, Zheng, Gong, Jiao, Li, Guo, Duan, Chen, et al., Luo et al.(2024)Luo, Yan, Hu, and Zhao], audio [Popov et al.(2021)Popov, Vovk, Gogoryan, Sadekova, and Kudinov, Liu et al.(2023)Liu, Chen, Yuan, Mei, Liu, Mandic, Wang, and Plumbley, Shen et al.(2023)Shen, Ju, Tan, Liu, Leng, He, Qin, Zhao, and Bian] and video [Khachatryan et al.(2023)Khachatryan, Movsisyan, Tadevosyan, Henschel, Wang, Navasardyan, and Shi, Blattmann et al.(2023)Blattmann, Rombach, Ling, Dockhorn, Kim, Fidler, and Kreis, Luo et al.(2023)Luo, Chen, Zhang, Huang, Wang, Shen, Zhao, Zhou, and Tan]. Furthermore, incorporating non-text conditions, such as video-to-audio [Du et al.(2023)Du, Chen, Salamon, Russell, and Owens, Luo et al.(2024)Luo, Yan, Hu, and Zhao], image-to-video [Gong et al.(2024)Gong, Zhu, Li, Kang, Wang, Ge, and Zheng, Ni et al.(2023)Ni, Shi, Li, Huang, and Min], and audio-to-image [Lee et al.(2023b)Lee, Kang, Kim, and Kim, Yang et al.(2023)Yang, Zhang, Ge, Shao, Xue, Qiao, and Luo] has led to impressive results. Notably, the integration of visual, textual, and auditory information as either condition or output [Tang et al.(2023b)Tang, Yang, Zhu, Zeng, and Bansal, Wu et al.(2023c)Wu, Fei, Qu, Ji, and Chua, Xing et al.(2024)Xing, He, Tian, Wang, and Chen] enables us to handle a broader range of tasks within a unified framework.

2.2 Diffusion Models in Editing

Building on the observation that DDIM sampling can be reversed to reconstruct or interpolate visual data [Song et al.(2020)Song, Meng, and Ermon], numerous studies have focused on diffusion-based editing. These works primarily target image editing with text prompts [Hertz et al.(2022)Hertz, Mokady, Tenenbaum, Aberman, Pritch, and Cohen-Or, Mokady et al.(2023)Mokady, Hertz, Aberman, Pritch, and Cohen-Or, Tumanyan et al.(2023)Tumanyan, Geyer, Bagon, and Dekel, Ju et al.(2023)Ju, Zeng, Bian, Liu, and Xu], aiming to achieve faithful inversion for enhanced editing capabilities. Additional control mechanisms have been incorporated alongside DDIM inversion, such as cross-attention [Hertz et al.(2022)Hertz, Mokady, Tenenbaum, Aberman, Pritch, and Cohen-Or], unconditional embeddings [Mokady et al.(2023)Mokady, Hertz, Aberman, Pritch, and Cohen-Or], and spatial and self-attention features [Tumanyan et al.(2023)Tumanyan, Geyer, Bagon, and Dekel].

Text-guided video editing has also drawn significant interest [Bar-Tal et al.(2022)Bar-Tal, Ofri-Amar, Fridman, Kasten, and Dekel, Wu et al.(2023a)Wu, Ge, Wang, Lei, Gu, Shi, Hsu, Shan, Qie, and Shou, Qi et al.(2023)Qi, Cun, Zhang, Lei, Wang, Shan, and Chen, Geyer et al.(2023)Geyer, Bar-Tal, Bagon, and Dekel], with an emphasis on preserving temporal consistency across edited frames. Moreover, the field of diffusion-based editing has expanded to include sound, which is utilized either as a prompt [Yang et al.(2023)Yang, Zhang, Ge, Shao, Xue, Qiao, and Luo, Lee et al.(2023a)Lee, Kim, Yoo, Yang, Cho, Kim, Chang, Kim, and Kim, Biner et al.(2024)Biner, Sofian, Karakaş, Ceylan, Erdem, and Erdem, Li et al.(2022)Li, Liu, Owens, and Zhao, Lin et al.(2025)Lin, Lin, Yang, Li, Wang, Lin, Wang, Bertasius, and Wang] or as the subject of editing [Liu et al.(2023)Liu, Chen, Yuan, Mei, Liu, Mandic, Wang, and Plumbley, Wang et al.(2024)Wang, Ju, Tan, He, Wu, Bian, et al., Manor and Michaeli(2024)].

Recently, it has been recognized that text prompts alone are insufficient for handling complex editing scenarios [Kumari et al.(2023)Kumari, Zhang, Zhang, Shechtman, and Zhu, Brack et al.(2024)Brack, Friedrich, Kornmeier, Tsaban, Schramowski, Kersting, and Passos, Zhao et al.(2024)Zhao, Chen, Chen, Bao, Hao, Yuan, and Wong, He et al.(2024)He, Zheng, Fang, Piramuthu, Bansal, Ordonez, Sigurdsson, Peng, and Wang]. To address this limitation, researchers have explored incorporating non-text conditions—such as edge map, segmentation mask, and depth map—to achieve richer and more precise control [Mo et al.(2024)Mo, Mu, Lin, Liu, Guan, Li, and Zhou, Zhao et al.(2024)Zhao, Chen, Chen, Bao, Hao, Yuan, and Wong, He et al.(2024)He, Zheng, Fang, Piramuthu, Bansal, Ordonez, Sigurdsson, Peng, and Wang, Lin et al.(2024)Lin, Cho, Zala, and Bansal]. However, most of these approaches expand the capabilities of diffusion models by adding and training additional modules to the text-to-image diffusion model, which requires extra training with paired datasets. CoDi-2 [Tang et al.(2023a)Tang, Yang, Khademi, Liu, Zhu, and Bansal] demonstrates instructional editing capability with multiple and multi-modal prompts, but requires extensive training on paired datasets of original data [Hertz et al.(2022)Hertz, Mokady, Tenenbaum, Aberman, Pritch, and Cohen-Or]. In contrast, our approach does not require training with paired data and has strong zero-shot editing ability.

3 Method

3.1 Background

A family of Latent Diffusion Model [Rombach et al.(2022)Rombach, Blattmann, Lorenz, Esser, and Ommer, Liu et al.(2023)Liu, Chen, Yuan, Mei, Liu, Mandic, Wang, and Plumbley] transforms a high-dimensional input data $x$ into a lower-dimensional latent $\mathbf{z}$ using the encoder $\mathcal{E}$ and decoder $\mathcal{D}$, where $\hat{x}=\mathcal{D}(\mathcal{E}(x))$. Subsequently, a time-conditioned U-net, denoted as $\theta$, conducts a sequential diffusion process to predict $\mathbf{z}_{0}$ from random noise $\mathbf{z}_{T}$, considering the textual condition $\mathcal{C}=\psi(\mathcal{P})$ as Equation 1, where $\psi$ refers to the prompt encoder and $\mathcal{P}$ is editing prompt.

[TABLE]

DDIM inversion [Song et al.(2020)Song, Meng, and Ermon] allows the original data to be encoded through an invertible transformation as expressed in Equation 2, resulting in initial noise $\mathbf{z}^{\ast}_{T}$ capable of reconstruction and interpolation. In editing tasks, conditional DDIM inversion with textual description \mathcal{P}_{\scalebox{0.7}{inv}} is commonly employed, where the initial noise $\mathbf{z}^{\ast}_{T}$ is utilized along with the editing prompt $\mathcal{P}$ in sampling stage to maintain the structure and semantic layout. We use $\ast$ to distinguish the diffusion trajectory of DDIM inversion and editing.

[TABLE]

However, this approach often leads to undesired changes [Mokady et al.(2023)Mokady, Hertz, Aberman, Pritch, and Cohen-Or], prompting the additional controls over the diffusion process for fine-grained editing [Couairon et al.(2022)Couairon, Verbeek, Schwenk, and Cord, Meng et al.(2021)Meng, He, Song, Song, Wu, Zhu, and Ermon, Mokady et al.(2023)Mokady, Hertz, Aberman, Pritch, and Cohen-Or, Tumanyan et al.(2023)Tumanyan, Geyer, Bagon, and Dekel]. PnP-Diffusion [Tumanyan et al.(2023)Tumanyan, Geyer, Bagon, and Dekel] has demonstrated the effectiveness of injecting spatial features and self-attention to preserve the original image content. Specifically, spatial features of the fourth decoder layer and the self-attention matrix are extracted during the DDIM inversion, which are then injected in the editing stage. TokenFlow [Geyer et al.(2023)Geyer, Bar-Tal, Bagon, and Dekel], which we adopt for video editing, adapts the PnP-Diffusion to video editing. It first randomly samples a subset of keyframes from the input video and jointly edits them via an extended self-attention mechanism as in Tune-A-Video [Wu et al.(2023a)Wu, Ge, Wang, Lei, Gu, Shi, Hsu, Shan, Qie, and Shou]. It then propagates the edited tokens from each keyframe’s self-attention map to its neighboring frames to ensure temporal coherence.

3.2 Training-Free Integration of Audio Prompts

With the rapid advancement of visual editing, research has evolved to integrate more diverse conditions beyond simple text prompts [Mo et al.(2024)Mo, Mu, Lin, Liu, Guan, Li, and Zhou, Zhao et al.(2024)Zhao, Chen, Chen, Bao, Hao, Yuan, and Wong, He et al.(2024)He, Zheng, Fang, Piramuthu, Bansal, Ordonez, Sigurdsson, Peng, and Wang, Lin et al.(2024)Lin, Cho, Zala, and Bansal, Tang et al.(2023a)Tang, Yang, Khademi, Liu, Zhu, and Bansal]. Most audio-conditioned approaches fine-tune the diffusion model to align sound with text [Yang et al.(2023)Yang, Zhang, Ge, Shao, Xue, Qiao, and Luo, Biner et al.(2024)Biner, Sofian, Karakaş, Ceylan, Erdem, and Erdem, Li et al.(2022)Li, Liu, Owens, and Zhao], but such adaptation couples the model to the specific training data and generalizes poorly to in-the-wild audio.

To overcome this limitation, we adopt a large-scale multimodal encoder such as ImageBind [Girdhar et al.(2023)Girdhar, El-Nouby, Liu, Singh, Alwala, Joulin, and Misra] and CoDi [Tang et al.(2023b)Tang, Yang, Zhu, Zeng, and Bansal]. Specifically, we utilize CoDi because (i) it relies on the same OpenCLIP-L/14 backbone as Stable Diffusion 1.x, common in visual-editing pipelines, and (ii) it is trained on the largest public audio corpus, giving robust audio representations.

However, integrating CoDi with Stable Diffusion is non-trivial, since they operate in different feature spaces. Stable Diffusion directly conditions on the CLIP text encoder’s output, while CoDi’s audio encoder produces embeddings in the aligned CLIP space. We denote the Stable Diffusion space by $\mathcal{C}_{\mathrm{SD}}$ and the aligned CLIP space by $\mathcal{C}_{\mathrm{CLIP}}$. Given a text prompt $\mathcal{P}$, the CLIP text encoder $\psi_{\mathrm{text}}$ generates an SD-space vector $\mathbf{c}=\psi_{\mathrm{text}}(\mathcal{P})\in\mathcal{C}_{\mathrm{SD}}$. Applying pooling to this output yields the special-token embedding $\mathbf{c}_{\mathrm{SD,pooled}}\in\mathcal{C}_{\mathrm{SD}}$. We then project and normalize:

[TABLE]

where $\mathbf{M}:\mathcal{C}_{\mathrm{SD}}\to\mathcal{C}_{\mathrm{CLIP}}$ is a learned linear mapping with no bias.

Given an audio embedding $\mathbf{c}_{A}\in\mathcal{C}_{\mathrm{CLIP}}$ from CoDi’s encoder $\psi_{\mathrm{audio}}$, we invert Eq. 3 and estimate its SD-space counterpart by solving a Tikhonov-regularized least squares problem:

[TABLE]

where $\mathbf{c}_{\mathrm{inv}}=\psi_{\mathrm{text}}(\mathcal{P}_{\mathrm{inv}})\in\mathcal{C}_{\mathrm{SD}}$, $\lambda=10^{-5}$, $P_{\mathrm{inv}}$ is the inversion prompt, and $I$ is the identity matrix. Finally, we replicate the inverted feature $\tilde{\mathbf{c}}_{\text{SD}}$ certain times and concatenate it with the first and last special tokens from $\mathbf{c}_{\mathrm{inv}}$. Despite minor precision errors that may arise from matrix inversion, the semantic content is effectively preserved in visual editing.

3.3 Separate Noise Branching with Adaptive Patch-wise Noise Selection

Multi-modal editing commonly requires multiple guidance signals—text, audio, depth, masks, and so on—to be combined within a single diffusion process [Mo et al.(2024)Mo, Mu, Lin, Liu, Guan, Li, and Zhou, Zhao et al.(2024)Zhao, Chen, Chen, Bao, Hao, Yuan, and Wong, He et al.(2024)He, Zheng, Fang, Piramuthu, Bansal, Ordonez, Sigurdsson, Peng, and Wang, Lin et al.(2024)Lin, Cho, Zala, and Bansal]. Existing works often enlarge the network or attach learned adapters, both of which incur additional training on specific paired dataset. We instead treat multi-prompt fusion as a variance-preserving aggregation of the model’s native noise estimates, yielding a parameter-free, training-free procedure that can be plugged into any pre-trained latent diffusion model.

Separate noise estimation. Let $\mathbf{z}_{t}\in\mathbb{R}^{c\times h\times w}$ be the latent at diffusion step $t$ and $\mathcal{C}=\{\mathcal{C}_{1},\dots,\mathcal{C}_{N}\}$ the set of $N$ encoded prompts (possibly of different modalities). For each prompt we query the frozen denoiser

[TABLE]

Limitations of naïve averaging. A naïve fusion, $\hat{\epsilon}_{\theta}=\tfrac{1}{N}\sum_{i}\epsilon_{i}$, reduces variance in proportion to $1/N$. Because diffusion process relies on scaled noise to reconstruct detail, this variance collapse produces suboptimal results, as observed in Fig. 4 (top) and quantified in Sec. 4.

Adaptive patch-wise Noise Selection. We utilize the inversion prompt $\mathcal{C}_{\text{inv}}$ and its noise prediction $\epsilon_{\text{inv}}=\epsilon_{\theta}(\mathbf{z}_{t},\,t,\,\mathcal{C}_{\text{inv}})$ to migitage this limitation. Subtracting $\epsilon_{\text{inv}}$ from each editing prompt yields residual feature maps $\Delta\epsilon_{i}=\epsilon_{i}-\epsilon_{\text{inv}}$, where each $\Delta\epsilon_{i}\in\mathbb{R}^{c\times h\times w}$ represents the difference in noise predictions for each prompt. Let $p$ index spatial locations on the $h\times w$ grid. We retain, for every patch, the residual with the largest magnitude:

[TABLE]

, where $\lVert\cdot\rVert$ is the channel-wise $\ell_{2}$ norm. Equation (6) preserves high-frequency detail and prevents destructive interference between heterogeneous conditions, as illustrated in Fig. 4 (bottom).

4 Experiments

In this section, we rigorously validate the effectiveness of our framework by constructing various evaluation datasets and baselines that incorporate audio editing prompts. Specifically, quantitative evaluations are conducted across four tasks: Audio-guided Image Editing (A2I-edit), Text and Audio-guided Image Editing (TA2I-edit), Audio-guided Video Editing (A2V-edit), and Text and Audio-guided Video Editing (TA2V-edit). Additionally, we explore more complex editing scenarios in Section 4.7.

The results from the A2I-edit and A2V-edit tasks assess whether our audio mapping can produce appropriate editing results while preserving the diverse information contained in the audio. The TA2I-edit and TA2V-edit tasks demonstrate that our noise integration method yields suitable results in complex editing scenarios that existing visual editing approaches cannot handle. The zero-shot capability of our framework can be further verified in the Sections 3, 4, 5, and 6 of the supplementary material.

4.1 Implementation Details

We incorporate diffusion editing techniques such as PnP-Diffusion [Tumanyan et al.(2023)Tumanyan, Geyer, Bagon, and Dekel] and TokenFlow [Geyer et al.(2023)Geyer, Bar-Tal, Bagon, and Dekel]. These techniques demonstrate superior edit fidelity and do not require word-level configurations, such as specifying which words to replace or emphasize. We employ Stable Diffusion v1.5 for fairness and utilize A100 with 40GB memory for inference with the seed set to 1 for all experiments.

4.2 Evaluation Dataset

To perform a comprehensive evaluation of our framework, we introduce PIEBench-multi and DAVIS-multi, which augment existing text-guided visual editing benchmarks, PIEBench [Ju et al.(2023)Ju, Zeng, Bian, Liu, and Xu] and DAVIS [Pont-Tuset et al.(2017)Pont-Tuset, Perazzi, Caelles, Arbeláez, Sorkine-Hornung, and Van Gool] to involve audio editing prompts. We select 28 classes of audio suitable for editing by filtering out low-quality content from VGGSound [Chen et al.(2020)Chen, Xie, Vedaldi, and Zisserman] and choose three audio clips for each category. We utilize 100 images in the PIEBench dataset and use all 89 videos from DAVIS, excluding one unsuitable video (gunshot). We then match appropriate audio editing prompts to each visual data, resulting in 300 editing pairs in PIEBench-multi and 267 editing pairs in DAVIS-multi. Detailed examples of our editing pairs used in the benchmarks can be found in Section 1 of the supplementary material.

4.3 Baselines

Complex audio-guided visual editing is underexplored and lacks baselines, so we first compare our method to representative text-guided approaches: Prompt-to-Prompt (P2P) [Hertz et al.(2022)Hertz, Mokady, Tenenbaum, Aberman, Pritch, and Cohen-Or], Null-Text Inversion (NT) [Mokady et al.(2023)Mokady, Hertz, Aberman, Pritch, and Cohen-Or], and PnP-Diffusion (PnP) [Tumanyan et al.(2023)Tumanyan, Geyer, Bagon, and Dekel] for image editing; and TokenFlow [Geyer et al.(2023)Geyer, Bar-Tal, Bagon, and Dekel] and Gen-1 [Esser et al.(2023)Esser, Chiu, Atighehchian, Granskog, and Germanidis] for video editing.

We initially consider a straightforward baselines that uses audio class names either alone (A2I-edit, A2V-edit) or appended to text editing prompts (TA2I-edit, TA2V-edit). Since this approach does not incorporate the textual description $\mathcal{P}_{\text{inv}}$ of original data, it induces severe modifications and produces suboptimal results. Section 2 of the supplementary material presents complete performances and examples for these baselines.

To better integrate audio information, we employ an audio captioning model [Kim et al.(2024)Kim, Jung, Lee, and Woo] to convert audio into text. Then we refine $\mathcal{P}_{\text{inv}}$ with audio caption using GPT-4o [Achiam et al.(2023)Achiam, Adler, Agarwal, Ahmad, Akkaya, Aleman, Almeida, Altenschmidt, Altman, Anadkat, et al.]. Section 2 of the supplementary material provides details on constructing these refined baselines, which we denote as X(caption).

For A2I-edit task, we compare our approach with AVStyle [Li et al.(2022)Li, Liu, Owens, and Zhao], Sonic Diffusion [Biner et al.(2024)Biner, Sofian, Karakaş, Ceylan, Erdem, and Erdem], and caption-based baselines. We exclude SGSIM [Lee et al.(2022)Lee, Roh, Byeon, Yoon, Kim, Kim, and Kim] as it is trained with highly specific domains and tends to produce inappropriate results on our benchmark. For TA2I-edit we include caption-based baselines, Sonic Diffusion, and 12 cascaded pipelines that sequentially run a T2I-edit (P2P, NT, or PnP) and then an A2I-edit (AVStyle or Sonic Diffusion). For A2V-edit and TA2V-edit tasks, we select baselines only using caption-based approaches (TokenFlow and Gen-1) since there are no suitable audio-guided video editing methods on our benchmarks.

4.4 Evaluation Metrics

To evaluate our framework, we employ metrics for content preservation and edit fidelity following existing works [Hertz et al.(2022)Hertz, Mokady, Tenenbaum, Aberman, Pritch, and Cohen-Or, Mokady et al.(2023)Mokady, Hertz, Aberman, Pritch, and Cohen-Or, Tumanyan et al.(2023)Tumanyan, Geyer, Bagon, and Dekel, Geyer et al.(2023)Geyer, Bar-Tal, Bagon, and Dekel]. For content preservation, we use Structure Distance [Tumanyan et al.(2022)Tumanyan, Bar-Tal, Bagon, and Dekel] and LPIPS [Zhang et al.(2018)Zhang, Isola, Efros, Shechtman, and Wang] for image editing tasks on PIEBench-multi (A2I-edit and TA2I-edit). For video editing tasks on DAVIS-multi (A2V-edit and TA2V-edit), we utilize frame-wise CLIP similarity (CLIP_frame) in accordance with the TGVE competition [Wu et al.(2023b)Wu, Li, Gao, Dong, Bai, Singh, Xiang, Li, Huang, Sun, et al.].

To evaluate edit fidelity, we apply CLIPScore [Hessel et al.(2021)Hessel, Holtzman, Forbes, Bras, and Choi] across all tasks. Specifically, we calculate CLIPScore for each individual editing prompt (CLIP_audio and CLIP_text) and use category names for audio editing prompts. However, in editing tasks with multiple prompts, it is crucial to evaluate how well each prompt is integrated and applied together. To evaluate this, we manually create integrated captions in text form for these tasks and report the results as CLIP_integrated.

Although these metrics assess the preservation of original data and the effectiveness of editing effects, they do not provide a comprehensive assessment. To address this limitation, we conducted a user study on Amazon Mechanical Turk with five participants per question, using a total of 100 samples for image editing tasks and all available samples for video editing tasks. Each participant compared our method against a baseline, selecting the better model according to structural preservation, audio alignment, text alignment, and overall condition consistency. To prevent cases where the absence of edits was rated highly for structural preservation, participants were instructed to give the lowest score for structural preservation if no edits were observed.

4.5 Evaluation Results

Table 1 shows the quantitative evaluation results of A2I-edit and TA2I-edit on PIEBench-multi, and Table 2 presents the quantitative evaluation results of A2V-edit and TA2V-edit on DAVIS-multi. We also show qualitative comparisons in Figure 2 and 3.

Audio-guided editing. Our model significantly outperforms the baselines in A2I-edit and A2V-edit in terms of CLIP_audio. Our method does not show competitive scores in content preservation metrics (Structure Distance, LPIPS, and CLIP_frame) since the baselines tend to preserve the original data excessively and do not well reflect the editing prompt.

Qualitative samples in Figure 2 and Figure 3 allow us to observe these characteristics in detail. In Fig. 2 (top), our method converts a bird to an airplane and adds a volcanic blast to a mountain, while baselines fail. In Fig. 3 (left), our model inserts the crowd following the audio prompt, whereas baselines scarcely change. More samples of the zero-shot capabilities of our framework can be found in Section 3 and Section 5 of the supplementary material. These results demonstrate that our framework effectively incorporates the rich information from audio and achieves diverse editing effects that are difficult to capture using text alone.

Audio and Text-guided editing. Similarly, our model outperforms the baselines in the CLIP_audio and CLIP_integrated. Baselines score well in CLIP_text but drop sharply in other metrics, showing that they capture only part of the desired edits in complex cases, a phenomenon also reported by [Kumari et al.(2023)Kumari, Zhang, Zhang, Shechtman, and Zhu, Brack et al.(2024)Brack, Friedrich, Kornmeier, Tsaban, Schramowski, Kersting, and Passos].

In Figure 2 (row 3), our method converts the dog into a rabbit and adds the splashing-water effect that baselines miss. In row 4, it morphs the woman into a blue-made-up baby, while baselines merely recolor the makeup. In Figure 3 (right) our model replaces the lady with Peppa Pig under rainfall, while baselines either omit the rain or over-distort the scene. Additional examples appear in Section 4 and Section 6 of the supplementary material, underscoring how separate-noise branching and adaptive patch selection effectively applies editing effects in complex scenarios.

User Study. Tables 1 and 2 present the results of the user study, demonstrating that our approach significantly outperforms the baselines in all cases, except for slightly lower performance than averaged noise prediction in terms of structural preservation. Although many baselines fail to produce any editing effects in several cases, users were instructed to account for such overly preserved images with the guideline: EXCEPTION: if an image is unchanged from the original image, consider it as the worst case. The results demonstrate that our method effectively edits images and videos in alignment with both text and audio while maintaining the original structure.

Note that in the main paper, only AVStyle $\rightarrow$ NT and NT $\rightarrow$ AVStyle were included among the 12 cascaded approach baselines for TA2I-edit, as these two methods demonstrate strong performance in A2I-edit and also due to space. We include the results of all baselines in Section 7 of the supplementary material. Also, we observe that CLIP_text reports higher scores than CLIP_audio since the text description of input data is included in the text editing prompt.

4.6 Ablation Study

To demonstrate the effectiveness of our noise integration approach, we conduct an ablation study. Specifically, we compare our method to one that integrates multiple noise predictions by simply averaging them. We report these results as ‘Ours (mean)’ in Tables 1 and Table 2. Simply averaging (mean) each noise prediction results in a drastic performance drop in CLIP_integrated scores across all tasks, which indicates that our separate noise branching and adaptive patch selection play a crucial role in handling complex multiple editing prompts.

Furthermore, Figure 4 visualizes the differences between the two methods when integrating multiple editing prompts. Specifically, we plot the magnitudes of the separate noise predictions from each editing prompt and the integrated noise at each diffusion timestep. As indicated by the red boxes, simply averaging the noise predictions causes the information from the audio editing prompt to be canceled out, resulting in the underwater bubbling effect not appearing in the final output.

4.7 Analysis and Discussions

Figure 5(a) illustrates the impact of modulating the magnitude of the audio, showing that an increase in magnitude enhances the editing effect. This demonstrates that our framework effectively integrates rich audio information such as magnitude, which is difficult to capture using text alone.

Figure 5(b) illustrates examples of more complex visual editing scenarios involving more than two editing prompts. We add text or audio editing prompts one at a time and visualize each result. We observe that our framework effectively determines which prompts should influence the objects and which should affect the background. Additional examples are provided in Section 8 of our supplementary material.

As for limitations, our frameworks inherit certain constraints from PnP-Diffusion and TokenFlow, particularly a tendency to preserve composition and content excessively or to yield flickering outputs. Although we did not address this in the present study, combining our framework with more powerful diffusion models and editing methods would also be worthwhile. Additionally, our mapping function assumes that audio features align closely with textual counterparts, which may not always be accurate [Liang et al.(2022)Liang, Zhang, Kwon, Yeung, and Zou]. Utilizing a variety of multi-modal encoders and diffusion models also appears to be a promising direction for future research.

5 Conclusion

In this paper, we introduce a framework for visual editing that seamlessly integrates audio conditions into diffusion models without requiring additional training. By leveraging a pre-trained multi-modal encoder with strong zero-shot capabilities, our approach mitigates the limitations of editing methods with only textual guidance, especially in complex scenarios where textual descriptions would be insufficient or ambiguous. We further introduce separate noise branching and adaptive patch selection, a novel method to handle multiple and multi-modal editing prompts without further training. Extensive experiments on newly established benchmark datasets demonstrate that our framework surpasses existing methods, effectively capturing rich audio information and coherently combining multiple prompts to produce sophisticated and accurate visual editing.

6 Acknowledgement

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2022-0-00608, Artificial intelligence research about multi-modal interactions for empathetic conversations with humans & No.RS-2020- II201336, Artificial Intelligence graduate school support(UNIST)). This work was also supported by National IT Industry Promotion Agency (NIPA) grant funded by the Korea government (MSIT) (No. PJT-25-033312, Development and Demonstration of a Fully Autonomous Speed Sprayer (SS) Based on On-Device AI).