Digital Twin-Empowered Deep Reinforcement Learning for Intelligent VNF Migration in Edge-Core Networks

TL;DR

This paper introduces a Digital Twin-enhanced deep reinforcement learning framework for optimizing virtual network function migration in edge-core networks, achieving lower latency and energy use through environment simulation and adaptive decision-making.

Contribution

It presents a novel DT-empowered DRL approach with a multi-task VAE and LSTM to improve training efficiency and migration performance in network management.

Findings

Significant reduction in end-to-end delay.

Notable decrease in energy consumption.

Faster policy convergence compared to traditional methods.

Abstract

The growing demand for services and the rapid deployment of virtualized network functions (VNFs) pose significant challenges for achieving low-latency and energy-efficient orchestration in modern edge-core network infrastructures. To address these challenges, this study proposes a Digital Twin (DT)-empowered Deep Reinforcement Learning framework for intelligent VNF migration that jointly minimizes average end-to-end (E2E) delay and energy consumption. By formulating the VNF migration problem as a Markov Decision Process and utilizing the Advantage Actor-Critic model, the proposed framework enables adaptive and real-time migration decisions. A key innovation of the proposed framework is the integration of a DT module composed of a multi-task Variational Autoencoder and a multi-task Long Short-Term Memory network. This combination collectively simulates environment dynamics and generates high-quality synthetic experiences, significantly enhancing training efficiency and accelerating policy convergence. Simulation results demonstrate substantial performance gains, such as significant reductions in both average E2E delay and energy consumption, thereby establishing new benchmarks for intelligent VNF migration in edge-core networks.

Tables

Table 1. Table I: DRL and DT parameters for simulation

Parameters	Values
Successful experience buffer size (${ℬ}_{SUC}^P$)	4000
Unsuccessful experience buffer size (${ℬ}_{FAIL}^P$)	2000
Virtual experience buffer size (${ℬ}^{DT}$)	6000
Balance parameter ($\varkappa$)	0.30
Learning rate of actor & critic	0.1
Discount factor ($\gamma$)	0.95
Physical mini-batch size ($M^P$)	32
DT mini-batch size ($M^{DT}$)	32
Optimizer	$Adam$

Equations

$$y_{r,v}^s(t)=\{\begin{array}{ll}1,& \text{server s serves VNF v at t}\\ 0,& \text{otherwise}.\end{array}$$

$$z_{r,\overline{l}}^l(t)=\{\begin{array}{ll}1,& \text{link l serves logical link lˉ at t}\\ 0,& \text{otherwise}.\end{array}$$

$$x^s(t)=\{\begin{array}{ll}1,& {\sum }_{r\in R}{\sum }_{v\in V_r}{y}_{r,v}^s(t)\ge 1,s\in \mu \\ 0,& \text{otherwise}.\end{array}$$

$$x^l(t)=\{\begin{array}{ll}1,& {\sum }_{r\in R}{\sum }_{\bar{l}\in \overline{L_r}}{y}_{r,v}^s(t)\ge 1,l\in \nu \\ 0,& \text{otherwise}.\end{array}$$

$$\sum _{r\in R}\sum _{v\in V_r}{y}_{r,v}^s(t)\cdot {\chi }_{r,v}^{CPU}(t)\le {\zeta }_s^{CPU},s\in \mu$$

$$\sum _{r\in R}\sum _{v\in V_r}{y}_{r,v}^s(t)\cdot {\chi }_{r,v}^{MEM}(t)\le {\zeta }_s^{MEM},s\in \mu$$

$$\sum _{r\in R}\sum _{\bar{l}\in \overline{L_r}}{z}_{r,\overline{l}}^l(t)\cdot {\chi }_{r,\overline{l}}^{BW}(t)\le {\zeta }_l^{BW},l\in \nu$$

$${\Phi }_r(t)=\{\begin{array}{ll}1,& {\sigma }_r^{ariv}\le t<\left({\sigma }_r^{ariv}+{\sigma }_r^{serv}\right),\forall t\in T,r\in R\\ 0,& \text{otherwise},\end{array}$$

$$D_{link}^l(t)=D_{trans}^l(t)+D_{prop}^l(t),l\in \nu ,r\in R$$

$$D_{proc}^s(t)=\frac{{\Gamma }_{CPU}^s(t)\cdot \tau }{1-{\Gamma }_{CPU}^s(t)},s\in \mu ,r\in R$$

$${\Gamma }_{CPU}^s(t)=\frac{\sum _{r\in R}\sum _{v\in V_r}{y}_{r,v}^s(t)\cdot {\chi }_{r,v}^{CPU}(t)}{{\zeta }_s^{CPU}},s\in \mu$$

$$\begin{array}{r}{D}_{E2E}^r(t)=\sum _{\bar{l}\in \overline{L_r},l\in L_r}{z}_{r,\overline{l}}^l(t)\cdot D_{link}^l(t)+\\ \sum _{v\in V_r,s\in \mu }{y}_{r,v}^s(t)\cdot D_{proc}^s(t)\end{array}$$

$$E_s(t)=y_{r,v}^s(t)\cdot \left({\epsilon }_{base}^s(t)+{\epsilon }_{cons}^s(t)\cdot {\Gamma }_{CPU}^s(t)\right)+{\epsilon }_{trans}^s(t),$$

$$\delta (t)=\frac{{\rm Y}-{{\rm Y}}^{\prime }}{{\rm Y}}$$

$$\eta (t)=\frac{\Lambda -{\Lambda }^{\prime }}{\Lambda }$$

$$\lambda =({\tau }_1\cdot \delta (t)+{\tau }_2\cdot \eta (t)),$$

$$\underset{\begin{array}{}\end{array}}{\max }\frac{1}{T}\sum _{t=1}^T\sum _{r\in R}\lambda ,\forall v\in V_r$$

$$\textbf{C1}:\text{Eq.}(5)-\text{Eq.}(7),\ \ \ \ \$$

$$\text{C2}:D_{E2E}^r(t)\le D_{max}^r,$$

$$L_{actor}({\Theta }^{\pi })=-{\mathbb{E}}_{\mathcal{S},\mathcal{A}}\left[\log \pi (\mathcal{A}|\mathcal{S};{\Theta }^{\pi })\cdot A(\mathcal{S},\mathcal{A})\right]$$

$$L_{critic}({\Theta }^V)={\mathbb{E}}_{\mathcal{S},\mathcal{R},{\mathcal{S}}^{\prime }}\left[{\left(\mathcal{R}+\gamma V({\mathcal{S}}^{\prime };{\Theta }^V)-V(\mathcal{S};{\Theta }^V)\right)}^2\right]$$

$$\mathcal{R}(\mathcal{S}(t),\mathcal{A}(t))=tanh(\lambda ),$$

$$\mathcal{V}(\mathcal{S}(t),\mathcal{A}(t))={\mathcal{S}}^{DT}(t),{\mathcal{A}}^{DT}(t)$$

$$\mathcal{L}(\theta ,\varphi )=$$

$$+\mathrm{KL}\left(q_{\varphi }(z|x_{in})\Vert p(z)\right),$$

$$\mathcal{T}({\mathcal{S}}^{DT}(t),{\mathcal{A}}^{DT}(t))={\mathcal{S}}^{DT}(t+1),{\mathcal{R}}^{DT}(t)$$

$${\mathcal{L}}_{MSE}=$$

$$+{\kappa }_2\cdot {\Vert {\mathcal{R}}^{DT}(t)-\mathcal{R}(\mathcal{S}(t),\mathcal{A}(t))\Vert }^2,$$

Full text

I Introduction

Network Function Virtualization (NFV) [12] marks a paradigm shift in network architecture, designed to lower operational costs by decoupling network functions (NFs) from specialized hardware appliances, such as conventional middle-boxes. By abstracting these NFs into software-based entities, namely, Virtualized Network Functions (VNFs) [3], NFV enables their deployment on general-purpose commodity servers. These deployments typically consist of ordered chains of VNFs including elements such as load balancers and firewalls. This architectural flexibility facilitates faster service provisioning, enhances scalability, optimizes resource utilization, and reduces operational expenditures for service providers [4].

However, as the adoption of NFV expands and the number of VNFs increases, scaling the VNF forwarding graph (VNF-FG) within edge networks becomes increasingly challenging due to their limited computational resources. In contrast, core networks typically possess abundant computing capacity [14], making them well-suited for offloading VNFs from the edge. Integrating edge and core infrastructures into a unified edge–core network architecture thus offers enhanced flexibility for optimizing end-to-end delay (E2E) and improving overall service performance [14]. Nevertheless, this shift toward large-scale, distributed architectures brings forth additional challenges, most notably, the need to ensure a stable and reliable power supply. Given that energy consumption accounts for a significant share of operational expenditure, its intelligent and adaptive management is essential for achieving sustainable network growth [5]. In response, Network Service Providers are actively pursuing strategies to reduce the energy consumption of these infrastructures, with the overarching goal of enhancing operational efficiency [8].

Recently, significant research efforts have focused on optimizing the scheduling of VNFs, particularly deployment and migration strategies [14, 6, 4] to minimize E2E delay [4] and reduce energy consumption [9, 5]. Various studies have proposed deployment strategies aimed at optimizing delay and energy consumption [4, 8]; however, these often rely on static VNF placement, assuming fixed network demands and unconstrained resource availability. Such assumptions undermine their applicability in real-world scenarios, where demand is dynamic and resource contention is common. To overcome these limitations, VNF migration [4, 2] has emerged as a more adaptive and robust solution. By enabling the flexible relocation of VNFs to more optimal servers while preserving application states, VNF migration allows network infrastructures to respond dynamically to changing workloads and resource conditions, ensuring service continuity and efficient resource utilization in heterogeneous environments [4].

In parallel, the increasing complexity and dynamism of modern networks have fueled interest in Machine Learning (ML) for intelligent network management [10]. Although widely used, supervised ML often requires labeled data and struggles with uncertainty. In contrast, many networking tasks involve sequential decision-making under uncertainty, making Reinforcement Learning (RL) a more suitable paradigm [6]. Specifically, RL has proven effective in optimizing the orchestration of VNF-FGs across heterogeneous infrastructures, particularly in response to fluctuating service demands. Building upon this foundation, Deep Reinforcement Learning (DRL)[6] combines RL with deep neural networks (DNNs) to tackle complex problems in high-dimensional spaces. DRL has shown great potential in tasks such as VNF migration in edge and core networks, where real-time decisions and efficient use of resources are crucial.

Although DRL has demonstrated notable performance improvements, it is typically implemented to solve large action spaces in tasks such as VNF migration and requires a substantial amount of training data. This, in turn, necessitates frequent interactions with the physical environment to collect sufficient data, imposing considerable overhead on real-world systems. To mitigate the burden a Digital Twin (DT) can be implemented to simulate environment dynamics and generate additional training data [15]. By serving as a virtual replica of the physical system, the DT enables the creation of a virtual experience buffer that, when combined with the physical experience buffer enriches the training dataset and accelerates policy convergence while reducing real-world exploration costs. Furthermore, hosting DRL training processes within the DT can offload computational workloads from the physical system, thereby reducing energy consumption and minimizing disruptions caused by continuous model training.

Building on the aforementioned insights, this study proposes a novel DT-empowered DRL framework for VNF migration in edge–core networks. The key contributions of this work are summarized as follows:

• •

To minimize the E2E delay of VNF-FGs and reduce the overall energy consumption of edge–core networks, this work proposes a DRL framework based on an Advantage Actor–Critic (A2C) architecture that intelligently migrates VNFs within the edge–core infrastructure.

• •

To enhance the performance and sustainability of the DRL framework for intelligent VNF migration, a novel integration with a DT is proposed that generates synthetic experience data and hosts the A2C training process. By shifting training from the physical environment to the DT, the framework significantly reduces reliance on real-world interactions and minimizes the energy cost and operational disruptions associated with continuous model updates.

• •

The DT module incorporates a multi-task Variational Autoencoder (VAE) as the generative model and a multi-task Long Short-Term Memory (LSTM) network as the predictive model. The generative model first produces simulated state–action pairs, and the predictive model then uses these to generate the corresponding simulated next states and reward values. Together, these components enrich the training dataset with synthetic transitions stored in the DT experience buffer, thereby improving the training efficiency of the A2C model.

• •

Extensive simulations demonstrate that the proposed framework significantly outperforms baseline methods. For example, in terms of average E2E delay and average energy consumption, the proposed framework achieves reductions of 7% and 15%, respectively, compared to baselines such as non DT-based A2C VNF migration.

II System Model and Problem Formulation

The considered edge–core network is modeled as a weighted undirected graph $\Omega=(\mu,\nu)$, where $\mu$ denotes the set of servers and $\nu$ represents the set of connectivity links between them. The server set is partitioned as $\mu=\mu_{e}\cup\mu_{c}$, where $\mu_{e}=\{s_{1},s_{2},\dots,s_{e}\}$ represents the subset of edge cloud servers, and $\mu_{c}=\{s_{e+1},\dots,s_{c}\}$ corresponds to the subset of core cloud servers. Here, $s_{1}\leq s\leq s_{e}$ indexes the edge cloud servers, and $s_{e+1}\leq s\leq s_{c}$ indexes the core cloud servers. The network architecture is assumed to include multiple levels of switching infrastructure to facilitate inter-server connectivity. Each link $l\in\nu$ connects a pair of distinct servers, such as $s_{e+1}$ and $s_{c}$, where $s_{e+1},s_{c}\in\mu_{c}$ and $s_{e+1}\neq s_{c}$. Each server $s\in\mu$ is characterized by its available computational resources, namely memory and CPU. Let $\zeta_{s}^{{MEM}}$ and $\zeta_{s}^{{CPU}}$ denote the memory and CPU capacities of server $s$, respectively. It is assumed that core cloud servers offer greater computational resources, whereas edge servers operate with comparatively limited capacity. Additionally, each connectivity link $l=(s_{e+1},s_{c})\in\nu$ has an associated available bandwidth denoted by $\zeta_{l}^{{BW}}$.

To accommodate dynamic network conditions, the NFV system is assumed to operate in a time-slotted manner. Each service request, represented as a VNF-FG $r\in R$, comprises a set of VNFs, denoted by $V_{r}=\{v_{1},v_{2},\dots,v,\dots,v_{p}\}$, with each VNF deployed on each $s\in\mu$ according to the strategy defined in [2]. At each time step $t$, each VNF $v$ has an associated resource demand characterized by memory, CPU, and bandwidth requirements. The memory and CPU demands of VNF $v$ in request $r$ are denoted by $\chi_{r,v}^{\text{MEM}}(t)$ and $\chi_{r,v}^{\text{CPU}}(t)$, respectively. In addition, the bandwidth requirement between any pair of connected VNFs, such as $v_{1}$ and $v_{2}$, is represented by $\chi_{r,\overline{l}}^{\text{BW}}(t)$, where $\overline{l}$ denote a logical link between the two VNFs and $\overline{l}\in\bar{L_{r}}$.

II-A VNF and Logical Link Mapping in VNF-FG, Associated Constraints, and Service Status

To begin with, two binary variables are introduced to represent the mapping status of VNFs and logical links within a VNF-FG $r$ at time step $t$.

[TABLE]

[TABLE]

Even if a server $s$ hosts at least one VNF, it must be activated to perform its functions; the same applies to a link $l$. Therefore, binary variables are introduced to indicate whether a server $s$ and a link $l$ are active at time step $t$.

[TABLE]

[TABLE]

To ensure that sufficient resources are available on both servers and links when servicing VNF-FGs, the following constraints are defined:

[TABLE]

[TABLE]

[TABLE]

Each VNF-FG $r\in R$ is associated with a service time $\sigma_{r}^{\text{serv}}$, representing the duration for which $r$ is expected to remain active, provided that all its constituent VNFs are successfully placed while satisfying all system constraints, otherwise, $\sigma_{r}^{{serv}}$ is set to zero. Accordingly, the service status of VNF-FG $r$ at time step $t$ is defined as:

[TABLE]

where, $\sigma_{r}^{ariv}$ denotes the arrival time of the VNF-FG $r$.

II-B E2E Delay Model

The E2E delay experienced by a VNF-FG consists of three main components: propagation delay, transmission delay, and processing delay. Drawing on the studies in [13] and [7] the delay incurred on a physical link $l$ for a VNF-FG $r$ at time step $t$ can be expressed as:

[TABLE]

here, $D^{l}_{{trans}}(t)$ and $D^{l}_{{prop}}(t)$ represent the transmission delay and propagation delay, respectively, at time step $t$.

Moreover, the processing delay at server $s$ at time step $t$ can be denote as:

[TABLE]

where, $\tau$ denotes the baseline processing delay per flow, and $\Gamma^{s}_{\text{CPU}}(t)$ is the CPU utilization of server $s$ at time step $t$, defined as:

[TABLE]

Consequently, the total E2E delay for VNF-FG $r$ at time $t$ can be calculated as:

[TABLE]

II-C Energy Consumption Model

The CPU is the primary source of server energy consumption, with its power usage assumed to scale linearly with utilization. Inspired by the energy consumption models presented in [1] and [9], the total energy consumption of server $s\in\mu$ that hosts VNF $v\in V_{r}$ from VNF-FG $r\in R$ at time step $t$ is given by:

[TABLE]

here, $\varepsilon^{s}_{{base}}(t)$ denotes the baseline energy consumption of server $s$, while $\varepsilon^{s}_{{cons}}(t)$ represents the incremental energy consumption per unit of CPU utilization, defined as, $\varepsilon^{s}_{{cons}}(t)$ = $\varepsilon^{s}_{{max}}(t)-\varepsilon^{s}_{{base}}(t)$, where, $\varepsilon^{s}_{{max}}(t)$ denotes the maximum energy consumption when server $s$ operates at full CPU capacity. In addition, $\varepsilon^{s}_{{trans}}(t)$ represents the transition energy consumption of server $s$ at time step $t$. This transition energy consumption accounts for the power overhead associated with changes in the server’s operational mode, such as transitioning from a sleep mode to an active mode due to VNF migration. If the operational mode of server $s$ remains unchanged between time steps $t-1$ and $t$, the transition energy consumption is assumed to be zero, that is, $\varepsilon^{s}_{{trans}}(t)=0$.

II-D Problem Formulation

The primary objective of this study is to maximize the reduction of both the E2E delay of VNF-FGs and the energy consumption in edge–core network through VNF migration. Consider, for example, a service request for 8K video streaming that is processed by a sequence of VNFs in a VNF-FG $r\in R$, such as a firewall (FW), traffic monitor (TM), and video optimization controller (VOC) at time step $t$. As illustrated in Figure 1, the E2E delay is reduced from 14 ms to 12 ms when the VOC is migrated from an edge cloud server $s_{e}$ to a core cloud server $s_{c}$. Furthermore, since edge cloud server $s_{e}$ no longer hosts any VNFs, it can be powered off, thereby contributing to a reduction in the overall energy consumption of the network.

The normalized reduction in E2E delay resulting from VNF migration is denoted as:

[TABLE]

where, $\Upsilon$ refers to the E2E delay before VNF migration, and $\Upsilon^{\prime}$ denotes the E2E delay after migration.

Similarly, the normalized reduction in energy consumption due to VNF migration is given by:

[TABLE]

where, $\Lambda$ represents the total energy consumption of the network before migration, and $\Lambda^{\prime}$ denotes the energy consumption after migration.

Since the objective is to jointly maximize the reduction in both E2E delay and energy consumption, a unified variable is defined by combining the two metrics with corresponding weighting coefficients. This composite variable, derived from Eq. (14) and (15), is expressed as:

[TABLE]

where, $\tau_{1}$ and $\tau_{2}$ are weighting coefficients in the range $[0,1]$.

Optimization 1. For a set of $R$ VNF-FGs over time steps $t=1$ to $T$, the VNF migration problem in the edge–core network is formulated as the maximization of the total average normalized reduction in E2E delay and energy consumption:

[TABLE]

subject to:

[TABLE]

[TABLE]

here, $D_{max}^{r}$ denotes the maximum tolerable E2E delay for VNF-FG $r$.

III DT-Empowered DRL for VNF Migration

III-A Key Procedures of the Proposed VNF Migration Scheme

The proposed VNF migration scheme operates as follows, with detailed descriptions of the DRL module and the DT module presented in the following subsections.

At the beginning of each time step $t$, the system scans all servers and links to remove timed-out VNF-FGs, i.e., those for which $\Phi_{r}(t)=0,\ \forall r\in R$. Subsequently, if there are incoming VNF-FG requests at time step $t$, the system accepts all of them and performs VNF deployment according to [2]. Following this, the system employs the DT-empowered DRL framework to make decisions regarding VNF migration when necessary. Finally, the network configuration and the set of accepted requests are updated, marking the active VNF-FGs with $\Phi_{r}(t)=1$.

III-B Description of the DRL Framework

Since Optimization 1 constitutes a joint optimization problem that aims to maximize the reduction of both E2E delay and average energy consumption, it poses significant challenges for conventional optimization techniques due to its complexity and inherent non-convexity. To address this, we reformulate Optimization 1 as a Markov Decision Process (MDP) and adopt the A2C algorithm to solve it effectively. The formulated MDP can be represented as $\mathcal{M}=\{\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P}\}$, which includes the state space $\mathcal{S}$, the action space $\mathcal{A}$, the reward function $\mathcal{R}$, and the state transition probability $\mathcal{P}$. In practice, state transition probabilities are typically unknown, and A2C addresses this by learning through direct interaction with the environment.

In A2C, two separate deep neural networks, that is, actor and critic are implemented. The actor network outputs a stochastic policy $\pi(\mathcal{A}|\mathcal{S};\theta^{\pi})$, while the critic estimates the state value function $V(\mathcal{S};\Theta^{V})$. The training leverages the advantage function defined as, $A(\mathcal{S},\mathcal{A})=\mathcal{R}+\gamma V(\mathcal{S^{\prime}};\Theta^{V})-V(\mathcal{S};\Theta^{V})$, which captures how much better an action $\mathcal{A}$ is compared to the expected value of the current state. This advantage term is used to guide policy updates, thereby reducing the variance of gradient estimates and accelerating convergence.

The actor loss is defined as:

[TABLE]

that encourages the policy to increase the probability of advantageous actions. The critic loss is computed as the mean squared error between the predicted and target value:

[TABLE]

This dual-network structure, combined with advantage-based updates, ensures stable and sample-efficient learning, making A2C well-suited for complex and high-dimensional optimization problems such as the one tackled in this study. The components of MDP, that is, state, action, and reward function are describe as follows:

**State: At time step $t$, the system state is represented by a vector $\mathcal{S}(t)$, consisting of eight key components: $\{\xi(t),\psi(t),\rho(t),\iota(t),\varphi(t),\varrho(t),\varpi(t),\varsigma(t)\}$. Specifically, $\xi(t)$ indicates whether each server is in active or sleep mode, while $\psi(t)$ indicates whether each server is hosting at least one VNF, where $\psi(t)=y_{r,v}^{s}(t)$. The components $\rho(t)$, $\iota(t)$, $\varphi(t)$, and $\varrho(t)$ represent the sets of CPU utilizations, memory utilizations, energy consumptions, and processing delays, respectively, for all servers $s\in\mu$ at time step $t$. Additionally, $\varpi(t)$ captures the server-level deployment mapping of VNFs for each VNF-FG $r\in R$, specifying which servers are hosting VNFs from $r$ at time $t$. Finally, $\varsigma(t)$ denotes the set of resource demands, specifically, the memory and CPU requirements associated with each VNF within its corresponding VNF-FG $r$.

Action: At time step $t$, the action taken by the system, denoted as $\mathcal{A}(t)$, represents the selection of servers from the set $\mu$ for migrating VNFs associated with a VNF-FG $r\in R$, where, $\mu=\mu_{e}\cup\mu_{c}$, with $\mu_{e}=\{s_{1},s_{2},\dots,s_{e}\}$ and $\mu_{c}=\{s_{e+1},\dots,s_{c}\}$.

Reward Function: At time step $t$, the reward function for migrating VNFs associated with a VNF-FG $r\in R$ is computed based on Eq. (16) as follows:**

[TABLE]

here, ${tanh}(\cdot)$ denotes the hyperbolic tangent function which is applied to normalize the reward within a bounded range.

III-C Description of the DT Module

In this study, a DT is established to enhance the performance and sustainability of the A2C model. Within the DT module, a multi-task VAE is implemented as the generative model to produce synthetic state–action pairs, and a multi-task LSTM network is utilized as the predictive model to estimate the corresponding next states and rewards. This combination enables the DT to emulate realistic environment dynamics and augment the learning process with high-quality synthetic data. Furthermore, by shifting the A2C training process from the physical environment to the DT, the framework significantly lowers the energy overhead and mitigates operational disruptions caused by frequent model updates.

At time step $t$, the generative model $\mathcal{V}$ takes as input the physical environment’s state $\mathcal{S}(t)$ and action $\mathcal{A}(t)$, and generates the corresponding state and action in the DT, denoted as $\mathcal{S}^{{DT}}(t)$ and $\mathcal{A}^{{DT}}(t)$. This process can be formally expressed as:

[TABLE]

The loss function for the generative model is composed of a reconstruction term and a regularization term. It can be expressed as:

[TABLE]

where, $x_{{in}}=(\mathcal{S}(t),\mathcal{A}(t))$ is the input to the encoder, $q_{\phi}(z|x_{{in}})$ is the approximate posterior, $p_{\theta}(\cdot|z)$ are the decoder likelihoods for state and action outputs, and $p(z)$ is the prior distribution over latent variables. The Kullback–Leibler (KL) divergence term ensures that the learned latent distribution remains close to the prior.

Moreover, at time step $t$, the predictive model $\mathcal{T}$ utilizes the generated DT state $\mathcal{S}^{{DT}}(t)$ and action $\mathcal{A}^{{DT}}(t)$ as inputs to predict the next state $\mathcal{S}^{{DT}}(t+1)$ and the corresponding reward $\mathcal{R}^{{DT}}(t)$, and can be expressed as:

[TABLE]

The adopted loss function for the predictive model is the mean squared error (MSE), can be formulated as:

[TABLE]

where, $\kappa_{1}$ and $\kappa_{2}$ are weighting coefficients, ranging from $[0,1]$.

A DT experience buffer, denoted as $\mathcal{B}^{DT}$ is established to store the generated data tuples, comprising $\langle\mathcal{S}^{{DT}}(t),\mathcal{A}^{{DT}}(t),\mathcal{R}(\mathcal{S}^{{DT}}(t),\mathcal{A}^{{DT}}(t)),\mathcal{S}^{{DT}}(t+1)\rangle$. This virtual experience buffer is utilized alongside the physical experience buffers to train the A2C, thereby improving sample efficiency and accelerating policy convergence.

III-D DT-Empowered DRL Framework Training

To train the A2C, the DT experience buffer $\mathcal{B}^{DT}$ and two physical experience buffers are utilized, namely, the successful physical experience buffer $\mathcal{B}^{P}_{{SUC}}$ and the unsuccessful physical experience buffer $\mathcal{B}^{P}_{{FAIL}}$. Separating successful and unsuccessful experiences helps the system distinguish between constraint-satisfying and constraint-violating actions, enabling more informed policy updates than conventional random sampling. At each time step $t$, after executing an action, if the constraints C1 and C2 are not satisfied, the corresponding experience is stored in $\mathcal{B}^{P}_{{FAIL}}$; otherwise, it is stored in $\mathcal{B}^{P}_{{SUC}}$. During each training iteration, the system randomly samples $\varkappa\cdot J^{P}$ experiences from $\mathcal{B}^{P}_{{SUC}}$ and $(1-\varkappa)\cdot J^{P}$ experiences from $\mathcal{B}^{P}_{{FAIL}}$ to form the physical mini-batch $M^{P}$. The parameter $\varkappa\in[0,1]$ controls the balance between successful and unsuccessful experiences and is critical for stable training. In the case of $\mathcal{B}^{{DT}}$, a DT mini-batch $M^{{DT}}$ is formed by randomly sampling $J^{{DT}}$ experiences from the $\mathcal{B}^{{DT}}$.

During the training iterations, the combined mini-batches $M^{P}$ and $M^{{DT}}$ are used to update the A2C actor and critic networks. After training, the updated actor parameters are transferred back to the physical environment for action selection. It is important to note that the generative and predictive models within the DT are trained using experiences drawn from $\mathcal{B}^{P}_{{FAIL}}$ and $\mathcal{B}^{P}_{{SUC}}$ at the beginning of each episode. The overall training and interaction process is illustrated in Figure 2.

IV Performance Evaluation

IV-A Simulation Setup

A physical edge–core network is generated based on the Waxman topology model with parameters $\alpha=0.5$ and $\beta=0.2$, following the configuration described in [11]. The network consists of 100 servers, with 35 designated as edge cloud servers and 65 as core cloud servers, as specified in [14]. The edge cloud servers are configured with 40 units of CPU resources and 16 units of memory resources, whereas the core cloud servers are configured with 200 units of CPU resources and 64 units of memory resources [14]. The CPU and memory utilization thresholds are both set to 80%. Additionally, the available bandwidth for each link is set to 3.5 Gbps [13]. The CPU and memory resource demands for each VNF within the VNF-FGs are randomly assigned, ranging from 1 to 20 units for CPU and from 1 to 4 units for memory, consistent with [14]. Each VNF-FG is composed of four VNFs. In each episode, 300 VNF-FG requests arrive sequentially, following a Poisson process with an arrival rate of 0.2 requests per time step.

In the A2C model, both the actor and critic networks are implemented as fully connected neural networks with four hidden layers comprising 1024, 512, 256, and 128 neurons, respectively, each using ReLU activation. The actor network generates a probability distribution over all candidate migration actions through a Softmax output layer. The critic network estimates the state-value using a single linear output neuron. In the DT module, the predictive model comprises a single LSTM layer with 64 hidden units, followed by a fully connected layer with 64 ReLU-activated neurons. Its output is split into two task-specific heads: one predicts the next state, the other the immediate reward, both using linear activation. The generative model includes an encoder with two ReLU-activated hidden layers of 64 and 32 neurons, followed by two output layers estimating the mean and log-variance of the latent distribution. The decoder consists of a shared hidden layer with 64 ReLU-activated neurons and two task-specific Sigmoid heads for reconstructing the digital twin state and action, respectively. Table I summarizes the simulation parameters used for the A2C and associated neural networks.

Furthermore, the Tensorflow-Keras platform is utilized to construct and train DNN models. Simulations are executed on a 2021 MacBook Pro, equipped with an Apple M1 Pro chip, featuring a 64-bit operating system, an 8-core CPU, a 14-core GPU, and 16GB of unified RAM.

For benchmarking performance comparisons, two VNF migration schemes are evaluated: (1) a non–DT-based A2C VNF migration and (2) a random VNF migration.

IV-B Numerical Results and Comparisons

Figure 3(a) illustrates that the proposed framework outperforms baseline approaches, such as the non–DT-based A2C VNF migration and random VNF migration scheme, in terms of average energy consumption across varying numbers of VNF-FGs. Specifically, it achieves reductions of 15% and 27%, respectively. This gain results from integrating A2C with the DT module, which provides synthetic constraint-aware transitions that improve the A2C’s ability to identify energy-efficient migration decisions. By exposing the policy to a broader range of system states, the DT helps avoid unnecessary server activations and accelerates convergence toward energy-minimizing behavior.

Figure 3(b) presents the average E2E delay of the proposed framework compared to baseline approaches, such as the non–DT-based A2C VNF migration and random VNF migration scheme. The proposed method achieves the lowest average E2E delay, with reductions of 7% and 18%, respectively. This improvement stems from the DT-generated constraint-aware synthetic transitions, which offer richer exposure to delay-critical states. Consequently, the policy avoids delay inducing actions such as migrating VNFs to overloaded servers and converges toward more delay-efficient migration strategies.

Figure 3(c) illustrates that the proposed framework achieves a higher average request acceptance ratio compared to both the non–DT-based A2C VNF migration and the random VNF migration scheme. This improvement is attributed to the DT–assisted learning process which enables the policy to anticipate resource contention and avoid infeasible migration decisions by leveraging synthetic, constraint-aware experiences, thereby maintaining higher service admission rates under dynamic network conditions.

V Conclusions

In this study, a novel DT-assisted DRL framework is proposed to enhance the efficiency of VNF migration in edge–core networks. By integrating the A2C model with a DT that captures network dynamics through a multi-task LSTM network and a multi-task VAE, the proposed solution effectively achieves the objective of minimizing both average end-to-end delay and energy consumption. The incorporation of synthetic experience buffers alongside physical experience buffers further improves sample efficiency, enabling robust training even under highly dynamic workload conditions. Extensive evaluation demonstrates that the proposed framework consistently outperforms traditional non–DT-based VNF migration and random VNF migration schemes across key performance metrics, such as average energy consumption, average E2E delay, and acceptance ratio. These findings validate the effectiveness of leveraging DT and DRL to enable scalable, adaptive, and sustainable orchestration of VNFs.