Lightning Fast Caching-based Parallel Denoising Prediction for Accelerating Talking Head Generation

TL;DR

This paper introduces LightningCP, a caching-based parallel denoising method, and DFA, a decoupled attention mechanism, to significantly accelerate talking head video generation without sacrificing quality.

Contribution

The paper presents a novel task-specific framework combining caching and decoupled attention to speed up diffusion-based talking head generation.

Findings

Significant inference speedup demonstrated in experiments.

High-quality videos maintained despite acceleration techniques.

Effective exploitation of spatial and temporal redundancies in videos.

Abstract

Diffusion-based talking head models generate high-quality, photorealistic videos but suffer from slow inference, limiting practical applications. Existing acceleration methods for general diffusion models fail to exploit the temporal and spatial redundancies unique to talking head generation. In this paper, we propose a task-specific framework addressing these inefficiencies through two key innovations. First, we introduce Lightning-fast Caching-based Parallel denoising prediction (LightningCP), caching static features to bypass most model layers in inference time. We also enable parallel prediction using cached features and estimated noisy latents as inputs, efficiently bypassing sequential sampling. Second, we propose Decoupled Foreground Attention (DFA) to further accelerate attention computations, exploiting the spatial decoupling in talking head videos to restrict attention to dynamic foreground regions. Additionally, we remove reference features in certain layers to bring extra speedup. Extensive experiments demonstrate that our framework significantly improves inference speed while preserving video quality.

Tables

Table 1. Table 1: Quantitative results on HDTF and MEAD. † We include 3 extra key timesteps for FasterDiffusion on EchoMimic to avoid severe video quality degradation.

Method	Variants	HDTF				MEAD				FLOPS $\downarrow$ ($\times {10}^{12}$/clip)	Latency $\downarrow$ (s/clip)	Speedup $\uparrow$
		FVD $\downarrow$	FID $\downarrow$	E-FID $\downarrow$	Sync $\uparrow$	FVD $\downarrow$	FID $\downarrow$	E-FID $\downarrow$	Sync $\uparrow$
Hallo	Base	135.32	4.69	0.157	7.38	153.42	6.17	0.297	6.03	2158.08	23.692	-
	DeepCache	129.31	4.62	0.157	7.52	137.56	6.26	0.276	6.15	1041.28	12.884	1.84
	FasterDiffusion	140.56	5.63	0.200	7.47	168.72	7.33	0.285	6.14	1561.64	9.032	2.62
	Ours	121.80	4.89	0.178	7.66	158.62	6.64	0.277	6.43	671.40	7.528	3.15
MEMO	Base	82.07	3.40	0.159	8.10	82.90	5.63	0.161	6.06	919.86	14.934	-
	DeepCache	82.52	3.42	0.162	8.17	85.28	5.66	0.170	6.09	491.14	9.212	1.62
	FasterDiffusion	98.12	4.17	0.232	8.23	93.49	5.89	0.221	6.24	712.54	6.740	2.22
	Ours	80.23	3.46	0.163	8.17	79.18	5.54	0.171	6.12	406.08	6.416	2.33
EchoMimic	Base	115.78	4.43	0.116	5.75	166.82	5.43	0.335	4.89	965.19	10.491	-
	DeepCache	165.42	4.57	0.093	5.86	234.59	6.65	0.384	4.83	437.97	5.460	1.92
	FasterDiffusion†	126.27	4.69	0.109	5.90	175.23	5.67	0.467	4.79	820.05	4.788	2.19
	Ours	116.94	4.55	0.103	6.08	195.57	5.45	0.391	4.93	286.77	3.387	3.10
EchoMimic (Acc.)	Base	145.35	5.06	0.196	6.27	213.72	6.27	0.361	5.33	90.27	1.0458	-
	DeepCache	146.56	5.10	0.203	6.40	214.52	6.15	0.367	5.49	67.49	0.8268	1.26
	FasterDiffusion	143.41	5.92	0.226	6.52	157.62	6.65	0.331	5.54	68.93	0.7680	1.36
	Ours	136.14	5.18	0.201	6.58	192.90	5.82	0.359	5.61	60.52	0.7236	1.45

Table 2. Table 2: Ablation study results on HDTF and MEAD. LCP : Lightning-fast Caching-based Parallel denoising prediction. DFA : Decoupled Foreground Attention. rm : reference feature removal.

Method	Variants	HDTF				MEAD				FLOPS $\downarrow$ ($\times {10}^{12}$/clip)	Latency $\downarrow$ (s/clip)	Speedup $\uparrow$
		FVD $\downarrow$	FID $\downarrow$	E-FID $\downarrow$	Sync $\uparrow$	FVD $\downarrow$	FID $\downarrow$	E-FID $\downarrow$	Sync $\uparrow$
Hallo	LCP	124.62	4.99	0.189	7.53	151.8	6.54	0.268	6.17	745.72	8.052	2.94
	LCP + DFA	128.31	5.10	0.198	7.57	154.48	6.79	0.274	6.19	679.56	7.688	3.08
	LCP + DFA + rm	121.80	4.89	0.178	7.66	158.62	6.64	0.277	6.43	671.44	7.528	3.15
MEMO	LCP	83.32	3.45	0.169	8.21	80.26	5.55	0.168	6.17	411.02	6.640	2.25
	LCP + DFA	80.49	3.43	0.162	8.16	79.57	5.50	0.168	6.11	406.82	6.500	2.30
	LCP + DFA + rm	80.23	3.46	0.163	8.17	79.18	5.54	0.171	6.12	406.08	6.416	2.33
EchoMimic	LCP	136.87	4.50	0.098	5.96	218.06	5.86	0.384	4.84	319.26	3.447	3.04
	LCP + DFA	116.33	4.64	0.099	5.98	180.79	5.56	0.388	4.94	291.21	3.405	3.08
	LCP + DFA + rm	116.94	4.55	0.103	6.08	195.57	5.45	0.391	4.93	286.77	3.387	3.10
EchoMimic (Acc.)	LCP	144.09	5.15	0.207	6.50	201.47	5.99	0.363	5.50	62.69	0.7368	1.42
	LCP + DFA	138.24	5.20	0.206	6.47	179.33	5.94	0.359	5.53	61.33	0.7326	1.43
	LCP + DFA + rm	136.14	5.18	0.201	6.58	192.90	5.82	0.359	5.61	60.52	0.7236	1.45

Table 3. Table 3: Ablation study results of input latents estimation on HDTF. w/ estimate : with input latents estimation. w/o estimate : without input latents estimation.

Method	Variants	HDTF
		FVD $\downarrow$	FID $\downarrow$	E-FID $\downarrow$	Sync $\uparrow$
Hallo	LCP w/o estimate	208.46	7.14	0.355	7.64
	LCP w/ estimate	200.33	7.04	0.352	7.63
EchoMimic	LCP w/o estimate	460.21	6.66	0.226	5.98
	LCP w/ estimate	225.16	6.22	0.233	6.05

Equations

$$z_t=\sqrt{{\bar{\alpha }}_t}{z}_0+\sqrt{1-{\bar{\alpha }}_t}ϵ,ϵ\sim \mathcal{N}(0,I)$$

$$z_{t-1}$$

$${\tau }_t$$

$$z_{t-1}$$

$${ϵ}_{t-1}$$

$$=U_{32}(f_{U_{31}}^t,\text{conv}(z_{t-1}),t-1,c)$$

$$z_{t-1-i}$$

$$\text{s.t.}1\le i\le N-1.$$

$${\hat{z}}_{t-i}$$

$$z_{t-1-i}$$

$$\text{s.t.}1\le i\le N-1.$$

$$A_{\mathcal{F}}=\text{softmax}\left(\frac{Q_{\mathcal{F}}{K}_{\mathcal{F}}^{\top }}{\sqrt{d}}\right)V_{\mathcal{F}}\in {\mathbb{R}}^{L_f\times d}.$$

$$A=\text{Merge}(A_{\mathcal{F}},A_{\mathcal{B}}^{(t^{\ast })})\in {\mathbb{R}}^{L\times d},$$

Full text

11footnotetext: Corresponding authors.

Introduction

Talking head generation has received significant attention due to its wide applications in virtual avatars, digital content creation, and real-time communication (Prajwal et al. 2020; Zhang et al. 2023; Wang et al. 2022; Tian et al. 2024). Recent advancements in generative diffusion models (Rombach et al. 2022; Tu et al. 2024) have greatly enhanced the realism and quality of talking head videos (Tian et al. 2024; Xu et al. 2024a; Chen et al. 2025; Zheng et al. 2024), surpassing earlier GAN-based techniques (Wang et al. 2021; Zhang et al. 2023; Wang et al. 2022). However, the computational cost associated with diffusion models—primarily due to iterative denoising steps and large parameter sizes—results in slow inference and hinders real-time deployment (Sun et al. 2025, 2024a). To accelerate diffusion models, previous methods employ model pruning (Fang et al. 2023) and distillation (Salimans and Ho 2022; Meng et al. 2023) techniques, but these often require expensive training.

More recently, DeepCache (Ma et al. 2024) proposes a training-free method that caches and reuses intermediate features to minimize redundant computations, achieving notable acceleration. FasterDiffusion (Li et al. 2023) reuses encoder features to parallelize decoding across multiple timesteps. However, these methods do not consider the unique redundancies in talking head generation, limiting their effectiveness in this task. To this end, we observed several critical redundancies present uniquely in talking head generation and propose a task-specific acceleration framework for maximal efficiency improvement. Fig. 1 illustrates high temporal redundancy in the input feature $f_{U_{31}}$ to the final upsampling layer $U_{32}$, as measured by both $L_{2}$ distance and cosine similarity, particularly at intermediate sampling stages. To exploit this, we propose a caching mechanism termed Lightning-fast Caching-based Parallel denoising prediction (LightningCP) that significantly improves inference speed without sacrificing quality. LightningCP caches and reuses $f_{U_{31}}$ across consecutive timesteps to bypass the entire encoder and most of the decoder, reducing model inference to merely a single upsampling layer. We further propose to parallelize the denoising predictions of these consecutive timesteps, significantly improving inference throughput.

Moreover, talking head videos consist of foreground portrait and background environment components, which are associated with unique attention patterns. Fig. 2(a) and 2(b) show that foreground tokens are highly correlated with each other while exhibiting low correlation with background tokens; In addition, Fig. 2(c), 2(d), and 2(e) demonstrate that the background components of the attention output features remain stable across many timesteps. Based on these insights, we propose Decoupled Foreground Attention (DFA) to perform attention exclusively with foreground tokens while reusing cached background output features, quadratically reducing the attention complexity with respect to foreground token ratio while maintaining generation quality.

In summary, we make the following contributions:

• •

We introduce a training-free acceleration framework for talking head generation, maximizing inference speed while maintaining or even improving generation quality.

• •

We propose Lightning-fast Caching-based Parallel denoising prediction (LightningCP), a caching mechanism that reduces UNet inference to a parallelizable upsampling layer by reusing temporally stable decoder features.

• •

We present Decoupled Foreground Attention (DFA) that reduces the computation complexity of attentions quadratically by leveraging the localization of foreground attention and the stability of background features.

• •

Extensive experiments demonstrate that our method outperforms existing caching-based acceleration methods in terms of both efficiency and quality.

Related Works

Diffusion-based Talking Head Generation

Diffusion-based talking head generation can be classified into two-stage and end-to-end methods. Two-stage methods (Ma et al. 2023; Liu et al. 2024; Li et al. 2025) predict a sequence of intermediate motion representations (e.g. 3DMM, implicit keypoints) from audio, and then generate talking videos from the predicted motion sequence using pretrained GAN-based generators. Although these methods can achieve real-time inference in some cases (Li et al. 2025), their generation quality is limited by the granularity of the motion representation and the rendering quality of the generator. In contrast, end-to-end methods (Tian et al. 2024; Xu et al. 2024a; Chen et al. 2025; Zheng et al. 2024) leverage the powerful generative priors of latent diffusion model (Rombach et al. 2022) and extend it with 3D convolutions, audio attentions, and temporal modules to generate talking head videos. While these methods can achieve higher generation quality, their slow inference speed hinders practical deployment. To bridge this gap, we propose a training-free approach for end-to-end models that yields substantial speedups while preserving quality, taking a significant step forward in real-world application.

Diffusion Model Acceleration

Diffusion models can be accelerated via efficient samplers (Song et al. 2020; Lu et al. 2022), model pruning (Fang et al. 2023), and distillation (Salimans and Ho 2022; Meng et al. 2023). However, efficient samplers do not exploit feature redundancy within diffusion models, while pruning and distillation incur expensive model training.

To address these limitations, training-free methods have gained popularity in recent years. ToMe (Bolya and Hoffman 2023) accelerates self-attention by removing less informative tokens, but cannot accelerate talking head generation significantly due to the more complex pipeline. DeepCache (Ma et al. 2024) caches high-level features and efficiently updates only low-level features, but requires inefficient iterative denoising. FasterDiffusion (Li et al. 2023) enables parallel decoding for multiple timesteps by reusing encoder features, but involves costly full decoder inference. In contrast, our cache reduces inference to only a single decoder layer and enables parallel denoising of multiple non-key timesteps with a novel input latents estimation technique.

Foreground-Background Decoupling

The decoupling of foreground (FG) and background (BG) has been used in various downstream tasks of diffusion models. FG-BG segmentation masks are used to preserve subject identity in consistent generation (Wang et al. 2025) and multi-prompt generation (Cai et al. 2025), as well as to preserve background in image and video editing (Zhu et al. 2025; Shuai et al. 2024; Sun et al. 2024b). In talking head generation, our method leverages the static nature of talking head background and employs masks to accelerate model inference for the first time.

Preliminaries

Latent Diffusion Models

Latent diffusion models (LDMs) (Rombach et al. 2022) have become the foundation for high-fidelity image and video generation. The diffusion process consists of two main stages: a forward process and a reverse process, both performed in the latent space. The forward process gradually adds Gaussian noise to a clean latent sample $z_{0}$, resulting in a sequence of increasingly noisy latents $\{z_{t}\}_{t=1}^{T}$:

[TABLE]

Here, $z_{t}$ denotes the noisy latent at timestep $t$, and $\epsilon$ is standard Gaussian noise. Let $\{\beta_{t}\}_{t=1}^{T}$ be a predefined noise schedule with $\beta_{t}\in(0,1)$, the signal scaling coefficient $\alpha_{t}$ is defined as $\alpha_{t}=1-\beta_{t}$, and the cumulative product $\bar{\alpha}_{t}$ is expressed as $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$.

The reverse process, parameterized by a neural network, aims to reconstruct the original latent $z_{0}$ from $z_{t}$ by iteratively removing noise. The DDIM sampling strategy is widely adopted for efficient and deterministic inference, with the reverse update at each timestep as:

[TABLE]

where $\epsilon_{\theta}(z_{t},t,c)$ is the predicted noise at timestep $t$, conditioned on $z_{t}$, the timestep $t$, and the condition vector $c$ that is typically a text prompt embedding.

Diffusion-based Talking Head Generation Model

Diffusion-based talking head generation models combine a backbone based on Stable Diffusion (Rombach et al. 2022) with additional modules to perform audio-driven animation. The backbone’s denoising UNet iteratively predicts and removes noise from the noisy latent guided by the reference feature and the audio feature. The reference feature is extracted by Reference Net (Xu et al. 2024b) from the input image to ensure identity consistency throughout denoising. To incorporate the reference feature, a reference attention module extends the self-attention of the denoising Unet by concatenating the noisy latent with the reference feature as key and value inputs. An audio attention module injects motion information into the denoising process by performing cross-attention between the noisy latent and audio feature, producing lip movements and head motions synced with the driving audio. A temporal layer performs self-attention on the frame dimension, capturing subtle transitions and enhancing frame-to-frame coherence. Finally, the denoised latent features are decoded by the VAE decoder into the talking head video frames.

At each timestep $t$ of the denoising process, the latent representation $z_{t}\in\mathcal{R}^{b,c,f,h,w}$ of a video clip is progressively refined, where $(b,c,f,h,w)$ represent batch size, channels, number of frames, image height, and image width. This iterative update can be formally expressed as follows:

[TABLE]

Specifically, the condition feature $c=\{R,A,c_{\text{others}}\}$ includes reference feature $R$, audio features $A$, and other customized condition features $c_{\text{others}}$ which may include face identity embedding (Xu et al. 2024a) or emotion features (Zheng et al. 2024). In the denoising process, the spatial attention modules concatenate the reference feature $R$ directly with noisy latent. Audio features $A$ spans the $f$ frames within the video clip. Finally, the noisy latent is reshaped from $b\times c\times f\times h\times w$ to $(b\times h\times w)\times f\times c$ as the input to the temporal layer. A key distinction between talking head generation and general video generation is that the former produces videos of varying lengths determined by the input audio clip duration. To accommodate this, the driving audio clip is segmented into fixed-length intervals, each corresponding to individual video clips of length $f$, which are then combined to form the complete video output.

Method

In this section, we detail the proposed acceleration methods for diffusion-based talking head generation. Our approach introduces Lightning-fast Caching-based Parallel denoising prediction (LCP) and Decoupled Foreground Attention (DFA), significantly reducing inference times while maintaining or even improving video quality. In addition, auxiliary techniques, including input latent estimation and reference feature removal, are proposed to further enhance the visual quality of specific diffusion-based talking head models.

Lightning-fast Caching-based Parallel Denoising Prediction

As shown in Fig. 3, diffusion-based talking head generation models typically adopt a UNet architecture, with four encoder (downsampling) blocks $D_{0},D_{1},D_{2},D_{3}$, a mid-block $M$, and four decoder (upsampling) blocks $U_{0},U_{1},U_{2},U_{3}$. In each denoising step, the input latent is passed sequentially through the encoder, mid-block, and decoder for denoising prediction. Since this full forward pass is repeated for every diffusion timestep, the process incurs substantial computation cost, especially with large latent diffusion models.

We observe that for most denoising timesteps in talking head generation, all downsampling operations in the encoder and the majority of the upsampling layers in the decoder are redundant. By caching and reusing high-level decoder features across timesteps, it is feasible to bypass the encoder entirely and all but the final upsampling layer of the decoder. As illustrated in Fig. 1, the input feature $f_{U_{31}}$ to the final layer $U_{32}$ of the last upsampling block exhibits substantial temporal similarity across consecutive denoising steps. Specifically, Fig. 1(a) depicts the feature $L_{2}$ distance between adjacent timesteps, while Fig. 1(b) presents the feature cosine similarity matrix computed across all timesteps. Both metrics confirm that $f_{U_{31}}$ maintains high temporal similarity, especially in the middle timesteps. This insight motivates us to cache $f_{U_{31}}$ at selected key timesteps and reuse it in subsequent non-key timesteps, significantly reducing computation cost without compromising the fidelity of the generated output.

Lightning-fast Cache

Motivated by this observation, we propose a lightweight caching strategy termed Lightning-fast Cache, which enables highly efficient inference by partially bypassing the U-Net architecture. Fig. 3 shows the pipeline of our proposed method. At each key timestep $t$, we execute a full forward pass of the UNet and cache the intermediate feature $f_{U_{31}}^{t}$. Then, for the following non-key timesteps $t-1$ and $t-2$, we skip all encoder and early decoder computations by reusing the cached $f_{U_{31}}^{t}$.

At non-key timesteps like $t-1$, the denoising network $\epsilon_{\theta}$ simplifies to a subnetwork $\hat{\epsilon}_{\theta}$, computing only the final upsampling operation $U_{32}$ using the cached $f_{U_{31}}^{t}$ and a lightweight convolution of the current latent $z_{t}$:

[TABLE]

where $\text{conv}(z_{t})$ denotes a shallow convolution applied to the latent $z_{t}$ to generate the skip-connection input for $U_{32}$, and $c$ denotes additional conditioning (e.g. audio feature, reference feature).

Parallel Denoising Prediction

With Lightning-fast Cache, we further accelerate denoising prediction of consecutive non-key timesteps in parallel on multiple GPUs, because inputs of these non-key timesteps share the same cached feature and differ only in their timestep-specific variables.

Given a key timestep $t$, we cache the intermediate feature $f_{U_{31}}^{t}$ and reuse it across a block of consecutive non-key timesteps $\{t-1,t-2,\cdots,t-N\}$. Then, the denosing equation of non-key timestep $t-i$ becomes:

[TABLE]

However, there is a key challenge in parallelization: when $i>1$, the input latents $z_{t-i}$ for each non-key timestep are not available in advance. A straightforward workaround is to approximate all unknown input latents using the latent $z_{t-1}$ computed at the key timestep.

Unfortunately, reusing the same input latent $z_{t-1}$ across multiple timesteps ${t-2,...,t-N}$ can lead to significant error accumulation, ultimately degrading video quality. As illustrated in Fig. 4(a), the feature similarity between consecutive input latents decreases substantially over time when using samplers like DDIM (Song et al. 2020). This compounding discrepancy highlights the risk of not updating the input latents during parallel denoising prediction.

To address this issue, we propose Input Latents Estimation, which estimates input latents for consecutive non-key timesteps in parallel prediction by leveraging the temporal feature similarity of the predicted noise. As illustrated in Fig. 4(b), the predicted noise remain stable across the timesteps where the input latents show increasing discrepancies. Specifically, we use the diffusion scheduler to estimate the input latents $\hat{z}_{t-2},...,\hat{z}_{t-N}$ for non-key timesteps $\{t-2,...,t-N\}$ using the predicted noise $\epsilon_{\theta}(z_{t},t,c)$ at key timestep $t$ through Eq. 6a, and then use them to compute the actual latents $\{z_{t-2},...,z_{t-N}\}$ via Eq. 6b. For timestep $t-2$, $\hat{z}_{t-2}$ is estimated using the latent $z_{t-1}$ computed at key timestep $t$; For subsequent timesteps after $t-2$, we iteratively estimate input latents $\hat{z}_{t-i}$ using the previously estimated latent $\hat{z}_{t-i+1}$ as the reference.

[TABLE]

This two-step refinement allows us to recover higher-quality output latents by using more accurate model inputs. In practice, we apply Input Latents Estimation after a designated threshold timestep $t_{\text{thresh}}$. In the early denoising phase ($t<t_{\text{thresh}}$), the input latent $z_{t}$ exhibits high similarity across steps, so parallel prediction without correction suffices. Beyond this threshold, the quality benefit of estimating $\hat{z}_{t-i}$ becomes significant. Our experiments confirm that Input Latents Estimation greatly improves video fidelity, particularly for high-speed samplers such as DDIM (Song et al. 2020), while incurring negligible extra computation.

Decoupled Foreground Attention

To further accelerate inference, we propose Decoupled Foreground Attention (DFA) to streamline the reference attention, self-attention in audio module, and temporal attentions in $U_{32}$. The computation cost of these attentions scales quadratically with the sequence length of the video tokens, which is as large as $64\times 64=4096$ in $U_{32}$. To reduce this cost, DFA restricts the attention computation to the foreground components of the input features. This reduction of spatial tokens decrease both query and key length, effectively reducing the attention complexity from $\mathcal{O}(L^{2})$ to $\mathcal{O}(L_{f}^{2})$, where $L_{f}$ is the number of foreground tokens.

Our design is based on two empirical observations:

• •

Foreground-Localized Attention: Foreground attention maps are primarily concentrated within the foreground region, indicating spatial decoupling between dynamic facial motion and static background content. Fig. 2(a) and 2(b) visualizes this phenomenon in the reference attention module of the upsampling layer $U_{32}$. Across all the foreground noisy latent tokens, we calculate the average attention weight sum per token with regard to tokens of 4 distinct groups: foreground noisy latent, background noisy latent, foreground reference feature, and background reference feature. The resulting analysis shows strong localization of attention within foreground tokens and weak correlation between foreground and background tokens.

• •

Temporal Redundancy in Background: Background attention features remain stable across consecutive timesteps, making them suitable for caching and reuse. Fig. 2 visualizes the $L_{2}$ distance for the background component of the attention output features in the upsampling layer $U_{32}$. The analysis on reference (Fig. 2(c)), audio (Fig. 2(d)), and temporal attentions (Fig. 2(e)) collectively reveal high feature similarity, particularly in the middle denoising timesteps, supporting the strategy of caching background features to reduce computational redundancy.

Foreground Masking for Token Reduction.

We use a $64\times 64$ face segmentation mask to separate the foreground tokens. The mask is downsampled from the reference image’s face segmentation mask, which is obtained using an off-the-shelf face parsing model (Yakhyokhuja 2024). Let $M\in\{0,1\}^{H\times W}$ denote the binary foreground segmentation mask, where an element value of $1$ indicates that the corresponding spatial location belongs to the foreground region. Given flattened attention features $Q,K,V\in\mathbb{R}^{L\times d}$, where $L=H\cdot W$, we select the foreground token subset as $Q_{\mathcal{F}},K_{\mathcal{F}},V_{\mathcal{F}}\in\mathbb{R}^{L_{f}\times d}$, where $L_{f}=\|M\|_{1}=\sum_{ij}M_{ij}$.

The attention output is then computed over the reduced foreground region:

[TABLE]

This leads to a quadratic reduction in attention complexity—from $\mathcal{O}(L^{2})$ to $\mathcal{O}(L_{f}^{2})$—proportional to the foreground token ratio. In reference attention, we apply the same foreground mask to both the noise and reference features. Similarly, in the audio and temporal attentions, the computation is restricted to foreground tokens.

Merging with Cached Background.

Once the updated foreground attention output $A_{\mathcal{F}}$ is obtained, we reconstruct the complete attention feature map by merging it with the cached background output $A_{\mathcal{B}}^{(t^{*})}\in\mathbb{R}^{L_{b}\times d}$ from the most recent key timestep $t^{*}$, where $L_{b}=L-L_{f}$ denotes the number of background tokens. The final combined feature is given by:

[TABLE]

By narrowing the expensive attentions to the foreground and reusing temporally stable background features, DFA achieves further acceleration with negligible impact on generation quality.

Reference Feature Removal

Reference attention incurs the highest computation cost in the entire pipeline because the contatenation of reference feature makes the key and value twice as long. Our empirical study shows that reference features in certain layers do not affect video quality, an effect that varies among different talking head models. In our experiments, we apply reference feature removal on all of the tested talking head models to achieve additional acceleration, while maintaining or even improving the video quality.

Experiments

Experimental Settings

Models, Datasets, and Evaluation Metrics

We evaluate our method on four diffusion-based talking head generation models: Hallo (Xu et al. 2024a), MEMO (Zheng et al. 2024), EchoMimic, and Accelerated EchoMimic (EchoMimic Acc.) (Chen et al. 2025), which accelerates inference by reducing the number of diffusion steps and removing classifier-free guidance (Ho and Salimans 2022). For testing, we use 100 videos from the HDTF dataset (Zhang et al. 2021) and 100 videos from the MEAD dataset (Wang et al. 2020).

To assess video quality, we use standard evaluation metrics for talking head generation: Fréchet Video Distance (FVD) (Unterthiner et al. 2018), Fréchet Inception Distance (FID) (Heusel et al. 2017), expression-FID (E-FID) (Tian et al. 2024), and Lip Sync (Prajwal et al. 2020). Specifically, E-FID quantifies the expressiveness of facial motion by computing FID between 3DMM (Deng et al. 2019) expression coefficients extracted from generated and ground-truth videos. To evaluate generation speed, we report latency as the UNet inference time required to generate a fixed-length video clip. Since the final video length varies with input audio duration, latency and FLOPs are measured per clip for fair comparison. Speedup is computed as the latency of the base model divided by the latency of the accelerated model.

Baselines

We compare our method with two training-free diffusion acceleration baselines: DeepCache (Ma et al. 2024) and FasterDiffusion (Li et al. 2023). For fair comparison, we use identical key timesteps across all acceleration methods for each model, as timestep selection substantially affects efficiency and output quality. All experiments are run using A800 GPUs. We use a single GPU to test latency of the base model and DeepCache, while using multi-GPUs to test latency of FasterDiffusion and our method.

Performance Analysis

Table 1 presents results on HDTF (Zhang et al. 2021) and MEAD (Wang et al. 2020) datasets, comparing our method with base models and acceleration baselines (Ma et al. 2024; Li et al. 2023) across multiple diffusion-based talking head models (Xu et al. 2024a; Zheng et al. 2024; Chen et al. 2025). Our method achieves the most inference acceleration, with the highest speedup on Hallo (3.15×) (Xu et al. 2024a) and EchoMimic (3.10×) (Chen et al. 2025). While MEMO (Zheng et al. 2024) and EchoMimic Acc. (Chen et al. 2025) have fewer diffusion steps—limiting the number of non-key timesteps—our method is still the fastest on every model. Furthermore, our approach achieves the lowest FLOPs in all settings, demonstrating superior computational efficiency via targeted optimizations: caching temporally stable decoder features and restricting high-cost attention operations to the dynamic foreground.

More importantly, this speedup is achieved without compromising visual quality. As shown in Fig. 5, our method maintains the visual fidelity, identity consistency, and facial movement expressiveness of the base model. In addition, it facilitates the generation of more synchronized lip movements compared to the base model and the acceleration baselines. On the HDTF dataset, our method consistently yields the lowest FVD across all models and the highest Sync scores in most cases, while maintaining FID and E-FID comparable to the best-performing baselines. On MEAD, our approach achieves the lowest FID and the highest Sync scores in most cases. These results highlight the effectiveness of the proposed mechanisms: re-using cached decoder feature in Lightning-fast Cache preserves the feature’s temporal coherency across timesteps, and Decoupled Foreground Attention preserves motion expressiveness. While other baselines may show competitive performance in isolated settings (e.g., DeepCache on Hallo or FasterDiffusion on EchoMimic Acc.), they often perform poorly on other models or datasets. In contrast, our method demonstrates consistent quality across all settings. Additionally, FasterDiffusion requires 3 extra key timesteps on EchoMimic to avoid severe video quality degradation, significantly diminishing its speedup. In comparison, our method retains both high video quality and high speed under the original settings, further validating its robustness and generalization ability.

Ablation Study

To assess the contribution of each component of our method, we conduct ablations on HDTF (Zhang et al. 2021) and MEAD (Wang et al. 2020) using three variants: (1) Lightning Fast Caching-based Parallel denoising prediction (LCP), (2) LCP + Decoupled Foreground Attention (DFA), and (3) full model with LCP + DFA + reference feature removal (rm), as shown in Table 2.

Adding LCP to the base model significantly accelerates inference speed while maintaining decent video quality. DFA reduces FLOPs and introduces additional speedup across all scenarios, notably achieving a 8.9% FLOPs reduction on Hallo and pushes speedup above 3. Additionally, it retains or even improves video quality, indicating that removing background tokens allow the model to preserve essential interactions within the face region. Reference feature removal further enhances lip sync, suggesting that eliminating redundant static appearance features in certain layers enables better modeling of fine-grained lip motion.

We also evaluate the effect of input latent estimation on a subset of HDTF. Table 3 shows that without this estimation, severe degradation in FVD, FID, and E-FID occurs, confirming that the drastic decrease in the similarity of input latent features across timesteps leads to significant error accumulation. Therefore, it is necessary to alleviate this error by estimating the input latents using the temporally stable predicted noise, which significantly improves generation quality with negligible computational overhead.

Conclusion

In this paper, we introduce Lightning-fast Caching-based Parallel denoising prediction (LightningCP), a training-free acceleration framework for diffusion-based talking head generation. By caching and reusing high-level decoder features, our approach enables efficient parallel inference across multiple denoising timesteps. Additionally, Decoupled Foreground Attention (DFA) further enhances inference efficiency by restricting attention to the foreground region. We also employ auxiliary techniques to improve generation quality via input latents estimation and introduce extra speedup via reference feature removal. Extensive experiment validates the superior efficiency and quality of our method, which provides a practical and plug-and-play solution for accelerating diffusion-based talking head generation.

Ethical Statement

This research is intended to support the development of ethical virtual humans and creative applications in entertainment and communication. However, as with other generative talking head technologies, there is a risk that the proposed acceleration framework could be misused for malicious purposes such as fraud, impersonation, or defamation (Liu et al. 2025). Moreover, the increasing prevalence of AI-generated media may contaminate public datasets and negatively impact the training of downstream AI models (Tu et al. 2025c, a), potentially leading to degraded or unsafe system behavior.

To mitigate these risks, we recommend that any generated talking head videos be clearly watermarked or labeled as AI generated. We are also committed to assist the ongoing community efforts to develop robust evaluations and safety protocols (Tu et al. 2025b), for promoting safe and trustworthy deployment of generative technologies.

Acknowledgements

This research is supported by the RIE2025 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) (Award I2301E0026), administered by A*STAR, as well as supported by Alibaba Group and NTU Singapore through Alibaba-NTU Global e-Sustainability CorpLab (ANGEL).