Continual Reinforcement Learning for Digital Twin Synchronization Optimization

TL;DR

This paper presents a continual reinforcement learning approach to optimize resource allocation for digital twin synchronization in dynamic wireless networks, significantly reducing state mismatch with efficient resource use.

Contribution

It introduces a novel CRL algorithm to solve a constrained Markov decision process for digital twin synchronization, enabling quick adaptation and improved accuracy.

Findings

CRL reduces NRMSE by up to 55.2% compared to traditional methods.

The approach adapts quickly to network capacity changes.

Efficient resource utilization in digital twin synchronization.

Abstract

This article investigates the adaptive resource allocation scheme for digital twin (DT) synchronization optimization over dynamic wireless networks. In our considered model, a base station (BS) continuously collects factory physical object state data from wireless devices to build a real-time virtual DT system for factory event analysis. Due to continuous data transmission, maintaining DT synchronization must use extensive wireless resources. To address this issue, a subset of devices is selected to transmit their sensing data, and resource block (RB) allocation is optimized. This problem is formulated as a constrained Markov process (CMDP) problem that minimizes the long-term mismatch between the physical and virtual systems. To solve this CMDP, we first transform the problem into a dual problem that refines RB constraint impacts on device scheduling strategies. We then propose a continual reinforcement learning (CRL) algorithm to solve the dual problem. The CRL algorithm learns a stable policy across historical experiences for quick adaptation to dynamics in physical states and network capacity. Simulation results show that the CRL can adapt quickly to network capacity changes and reduce normalized root mean square error (NRMSE) between physical and virtual states by up to 55.2%, using the same RB number as traditional methods.