Task-Oriented Edge-Assisted Cross-System Design for Real-Time Human-Robot Interaction in Industrial Metaverse

TL;DR

This paper introduces a task-oriented edge-assisted framework using digital twins for real-time human-robot interaction in industrial Metaverse, improving responsiveness, accuracy, and visual fidelity under challenging conditions.

Contribution

It proposes a novel cross-system framework with a meta-learning algorithm for dynamic prediction, enhancing real-time interaction and adaptability in industrial environments.

Findings

Significantly reduces trajectory prediction error.

Achieves high visual fidelity in 3D scene rendering.

Demonstrates effective real-time interaction in industrial tasks.

Abstract

Real-time human-device interaction in industrial Metaverse faces challenges such as high computational load, limited bandwidth, and strict latency. This paper proposes a task-oriented edge-assisted cross-system framework using digital twins (DTs) to enable responsive interactions. By predicting operator motions, the system supports: 1) proactive Metaverse rendering for visual feedback, and 2) preemptive control of remote devices. The DTs are decoupled into two virtual functions-visual display and robotic control-optimizing both performance and adaptability. To enhance generalizability, we introduce the Human-In-The-Loop Model-Agnostic Meta-Learning (HITL-MAML) algorithm, which dynamically adjusts prediction horizons. Evaluation on two tasks demonstrates the framework's effectiveness: in a Trajectory-Based Drawing Control task, it reduces weighted RMSE from 0.0712 m to 0.0101 m; in a real-time 3D scene representation task for nuclear decommissioning, it achieves a PSNR of 22.11, SSIM of 0.8729, and LPIPS of 0.1298. These results show the framework's capability to ensure spatial precision and visual fidelity in real-time, high-risk industrial environments.

Tables

Table 1. TABLE I: System Parameters for Performance Evaluation

Prototype Setup		Learning Setup
Parameters	Values	Parameters	Values
Input device sampling frequency	$120\mathrm{Hz}$	Step length of one-step gradient update $\alpha$	$1\text{×}{10}^{-3}$
Input length of prediction function $W_p$	$4000\mathrm{ms}$	Meta-step length $\beta$	$1\text{×}{10}^{-5}$
Prediction horizon of prediction function $H$	$1000\mathrm{ms}$	Batch size	256
Simulation frequency $1/T_{si}$	$240\mathrm{Hz}$	Discount factor $\gamma$	$0.99$
Rendering step length $T_{im}$	$16\mathrm{ms}$	General advantage estimation parameter $\lambda$	$0.99$
Rendering resolution	$1920\text{px}\times 1080\text{px}$	Clip range	0.2
Real-world robotic arm sampling frequency	$1000\mathrm{Hz}$	Weighting coefficient ${\omega }_1,{\omega }_2,{\omega }_3,{\omega }_4$	-1

Table 2. TABLE II: Measured E2E delays and the number of convergence episode

Communication delay (ms)	$T_d=0$	$T_d\sim 𝒩(50,{10}^2)$	$T_d\sim 𝒩(100,{10}^2)$
Average RMSE (m)	0.0098	0.0094	0.01435
Convergence episode number	220	350	540
Average E2E delay (the Metaverse) (ms)	29.1	127.4	236.2
Average E2E delay (real-world workspace) (ms)	35.5	143.8	251.7

Table 3. TABLE III: Evaluation of Performance for different tasks

RMSE	Pose	HITL-MAML	”WP”	”OD”	”RS”
Circle	Position-vitual (m)	0.0091	0.0694	0.0398	0.0195
	Orientation-vitual	0.0086	0.0475	0.0445	0.0223
	Position-real (m)	0.0098	0.0812	0.0413	0.0264
	Orientation-real	0.0104	0.0546	0.0476	0.0211
Square	Position-vitual (m)	0.0067	0.0612	0.0365	0.0265
	Orientation-vitual	0.0068	0.0461	0.0278	0.0189
	Position-real (m)	0.0089	0.0689	0.0374	0.0210
	Orientation-real	0.0089	0.0691	0.0377	0.0227
Star	Position-vitual (m)	0.0099	0.0649	0.0514	0.0342
	Orientation-vitual	0.0123	0.0832	0.0446	0.0258
	Position-real (m)	0.0137	0.0792	0.0481	0.0298
	Orientation-real	0.0121	0.0711	0.0475	0.0243
Triangle	Position-vitual (m)	0.0106	0.0715	0.0427	0.0279
	Orientation-vitual	0.0114	0.0692	0.0392	0.0332
	Position-real (m)	0.0125	0.0811	0.0479	0.0295
	Orientation-real	0.0118	0.0732	0.0401	0.0224

Table 4. TABLE IV: Comparative Analysis of Proposed algorithm Performance for Real-time 3D scene representation

	Fixed Photographing number - 10				Fixed Task Time - 20 s
Approaches	HITL-MAML	”WP”	”OD”	”RS”	HITL-MAML	”WP”	”OD”	”RS”
Average Task Time (s)	20.4	34.2	29.6	28.1	———————————————
Average Photo Taken	———————————————				10	5	7	7
Average RMSE (m)	0.0116	0.0639	0.0395	0.0312	0.0120	0.0758	0.0413	0.0298
PSNR	22.11	21.94	22.03	22.49	23.81	19.46	23.21	22.84
SSIM	0.8729	0.8465	0.8647	0.8701	0.8854	0.8011	0.8753	0.8677
LPIPS	0.1298	0.1443	0.1340	0.1295	0.1171	0.2007	0.1233	0.1341

Equations

$$T_r=t_9-t_1.$$

$$T_v=t_{10}-t_1.$$

$$[{\hat{\mathbf{p}}}_r(t_1),$$

$$\hat{\mathbf{p}}(t_1)=c+\sum _{a=1}^p{\varphi }_a\mathbf{p}(t_1-a)+\sum _{b=1}^q{\theta }_bϵ(t_1-b),$$

$$[H_r(t_1),H_v(t_1)]={\pi }_{\theta }([\mathbf{p}(t_1),T_r(t_1),T_v(t_1)]),$$

$${\tilde{\mathcal{T}}}_r(t_2)={\mathcal{F}}_r({\hat{\mathbf{p}}}_c(t_2),{\sigma }_v(t_2),{\sigma }_r(t_2),{\theta }_c),$$

$${\ddot{\mathbf{q}}}_i(t_2)=k_p\cdot \mathcal{R}({\mathbf{q}}_i^{\prime }(t_2)-{\mathbf{q}}_i(t_2))-k_d\cdot {\dot{\mathbf{q}}}_i(t_2),$$

$$\mathcal{R}(\mathbf{u}(t_2))=\{\begin{array}{ll}\mathbf{u}(t_2),& \text{if }||\mathbf{u}(t_2)||<{\theta }_{th},\\ {\theta }_{th}\cdot \frac{\mathbf{u}(t_2)}{||\mathbf{u}(t_2)||},& \text{otherwise}.\end{array}$$

$${\hat{\mathbf{q}}}_i(t_3)=\{\begin{array}{ll}{\alpha }_s\cdot {\hat{\mathbf{q}}}_i(t_3-T_{co})+(1-{\alpha }_s)\cdot {\mathbf{q}}_i(t_3),& t_e<t_d\\ {\hat{\mathbf{q}}}_i(t_3-T_{co}),& \text{otherwise}.\end{array}$$

$${\tilde{\mathcal{T}}}_v(t_4)={\mathcal{F}}_v({\hat{\mathbf{p}}}_r(t_4),{\sigma }_v(t_4),{\sigma }_r(t_4),{\theta }_v),$$

$${\tilde{\mathcal{T}}}_s(t_5)={\mathcal{F}}_u({\tilde{\mathcal{T}}}_v(t_5),{\theta }_u,{\theta }_e),$$

$${\mathbf{I}}_v(t_6)={\mathcal{F}}_{re}({\tilde{\mathcal{T}}}_s(t_6),{\theta }_{re}),$$

$${\tilde{\mathcal{T}}}_{rw}(t_7)={\mathcal{F}}_{rw}({\tilde{\mathcal{T}}}_{ri}(t_7),{\theta }_{rw}),$$

$$u_i(t_7)=K_p{\tau }_i(t_7)+K_i{\int }_0^{t_7}{\tau }_i(\dot{t})\mathbf{d}\dot{t}+K_d\frac{\mathbf{d}}{\mathbf{d}t}{\tau }_i(t_7),$$

$$\left[\begin{array}{c}{l}_{x,i}\\ l_{y,i}\\ l_{z,i}\end{array}\right]=s\cdot R_p\cdot \left[\begin{array}{c}{l}_x\\ l_y\\ l_z\end{array}\right]+d$$

$$e_v={\omega }_1\cdot {\text{RMSE}}_p({\tilde{p}}_i,{\tilde{p}}_v)+{\omega }_2\cdot {\text{RMSE}}_o({\tilde{p}}_i,{\tilde{p}}_v)$$

$$e_r={\omega }_3\cdot {\text{RMSE}}_p({\tilde{p}}_i,{\tilde{p}}_r)+{\omega }_4\cdot {\text{RMSE}}_o({\tilde{p}}_i,{\tilde{p}}_r)$$

$$\mathcal{M}={\mathcal{F}}_{\text{NeRF}}(\{{\mathbf{I}}_v(t_i)\})$$

$$\text{KPI}(\mathcal{M})=\{\text{PSNR}(\mathcal{M}),\text{SSIM}(\mathcal{M}),\text{LPIPS}(\mathcal{M})\}$$

$$r({\mathbf{s}}_t,{\mathbf{a}}_t)={\mathbf{e}}_v(t)+{\mathbf{e}}_r(t),$$

$${\rho }_t^{[i]}=\left(\begin{array}{c}\mathrm{Pr}\{a_t^{[i]}=H\}\\ ⋮\\ \mathrm{Pr}\{a_t^{[i]}=1\}\\ \mathrm{Pr}\{a_t^{[i]}=0\}\end{array}\right),i=\{1,2\},$$

$$T_n=\{{\mathcal{L}}_{T_n}({\pi }_{\theta }),\{({\mathbf{a}}_{t_1},{\mathbf{s}}_{t_1}),...,({\mathbf{a}}_{t_L},{\mathbf{s}}_{t_L})\},{\rho }_t,L\},$$

$${\mathcal{L}}_{T_n}($$

$$\text{clip}\left(\frac{{\pi }_{\theta }({\mathbf{a}}_t\mid {\mathbf{s}}_t)}{{\pi }_{{\theta }^k}({\mathbf{a}}_t\mid {\mathbf{s}}_t)},1-ϵ,1+ϵ\right)A^{{\pi }_{{\theta }_t}}({\mathbf{s}}_t,{\mathbf{a}}_t)),$$

$$A^{{\pi }_{\theta }}({\mathbf{s}}_t,{\mathbf{a}}_t)=Q^{{\pi }_{\theta }}({\mathbf{s}}_t,{\mathbf{a}}_t)-V^{{\pi }_{\theta }}({\mathbf{s}}_t),$$

$${\theta }_n=\theta -\alpha \cdot {\nabla }_{\theta }{\mathcal{L}}_{T_n}({\pi }_{\theta }),$$

$${\theta }^{\prime }\leftarrow \theta -\beta \cdot {\nabla }_{\theta }\sum _{n=1}^N{\mathcal{L}}_{T_n}({\pi }_{{\theta }_n}),$$

$$\underset{\theta ,H_v(t),H_r(t)}{\mathrm{argmin}}$$

$${\mathcal{L}}_{T_n}(\theta )=\mathbb{E}$$

$$0<H_v(t)<H_v$$

$$0<H_r(t)<H_r$$

Full text

I Introduction

Industrial Metaverse represents an integrated virtual ecosystem that extends the concept of the Metaverse to specific industrial sectors, merging physical and digital realms. It explores the transformative potential of teleoperation, real-time collaboration, and synchronization within high-risk industries, driving substantial advancements in industrial operations [1]. Digital twins (DTs) are a key enabler within the larger framework of industrial Metaverse, facilitating real-time data interaction and providing highly accurate virtual models of physical assets [2]. For instance, in nuclear decommissioning, industrial Metaverse, aided by DTs, can facilitate real-time remote teleoperation to inspect critical components like internal wall tiles in fusion reactors. This enables precise monitoring and safe handling during dismantling, ensuring both safety and efficiency [3] [4].

Despite the advanced capabilities of industrial Metaverse, significant challenges remain in realizing its full potential. One of the fundamental issues lies in the hidden and dynamic nature of latency within highly coupled cyber-physical subsystems [5]. Unlike conventional networks where latency can be measured and managed in relative isolation, in industrial Metaverse environments latency emerges across multiple tightly interconnected subsystems (e.g., sensing, control, rendering, and actuation). This interdependence makes latency not only more difficult to analyze but also highly dynamic, since variations in one subsystem may propagate and amplify through others. Such complexity implicitly calls for cross-system design principles rather than subsystem-level optimization alone [6].

Compounding this challenge, the requirements of latency and reliability in industrial Metaverse are still poorly understood. Existing Key Performance Indicators (KPIs), such as those established for ultra-low latency, high reliability, and high bandwidth, provide necessary technical baselines; however, they do not directly capture the task-level performance demands of industrial Metaverse applications. For instance, [7] defines fine-grained KPIs for video streaming in industrial Metaverse scenarios, suggesting that data rates of 1 Gbps (for smooth playback) or 2.35 Gbps (for interactive experiences) are required to support 8K resolution, 120 FPS, and 360-degree video streaming [8]. Similarly, the Tactile Internet prioritizes ultra-low latency to ensure that tactile sensations and control commands are transmitted almost instantaneously, with latency requirements often below 1 ms [9], as humans are more sensitive to delays in tactile feedback than to audio or visual delays. This requirement is critical in applications where even slight feedback delays may cause errors or degraded performance, particularly in scenarios involving human-machine interaction or fine motor control [10].

Nevertheless, these technical metrics remain an imperfect proxy for actual task demands. Tasks in industrial Metaverse are often diverse and complex, ranging from collaborative teleoperation to predictive maintenance, each with distinct sensitivity to communication impairments [11]. Simply optimizing communication against conventional KPIs may therefore be suboptimal, as these metrics act like the “shortest stave in the barrel”: the system must meet the most stringent requirement even if it is irrelevant to the task at hand. This mismatch reveals a critical gap between traditional communication metrics and task-specific requirements, motivating the development of task-oriented communication frameworks where communication resources are allocated and optimized directly toward task success [12]. Furthermore, to reduce dependence on communication bottlenecks, edge deployment of virtual world components and decision-making modules is emerging as a promising direction [13]. By enabling intelligent decision-making closer to the physical processes, edge computing can reduce end-to-end latency and enhance real-time interaction between human operators, robots, and digital twins [14]. However, what functions should be deployed at the edge, how they should be partitioned between the edge and cloud, and how intelligent coordination can be achieved across heterogeneous nodes remain open and unresolved questions. Addressing these challenges requires not only innovations in networking protocols but also integrated approaches spanning communication, computation, and task-level system design.

Furthermore, another significant challenge in industrial Metaverse is the intricate complexity of human-in-the-loop (HITL) systems, where seamless interaction between human operators and automated systems is critical yet profoundly challenging. In these scenarios, delays in control commands and feedback loops can significantly affect system performance and safety [15]. For instance, in smart grids, where human operators manage and control critical infrastructure, latency in decision-making can lead to suboptimal responses to dynamic conditions such as load changes or fault detection [16]. While research on the effects of latency in teleoperation provides a foundational understanding [17], the intricacies of human behavior and interaction with complex industrial Metaverse environments require more sophisticated models and simulations to predict and mitigate these effects. This presents a significant challenge for deploying Artificial Intelligence (AI)-driven control systems that can adapt to human behavior and ensure seamless, real-time operation.

I-A Related Work

1. Edge-Assisted System for Industrial Metaverse: By balancing the different communication, computation, and control capabilities of different devices and offloading some or all of the tasks to the edge or the cloud, collaborative computing and sharing of multidimensional communication, caching, and computational resources including the large language model (LLM) can be efficiently achieved in industrial Cyber-Physical System (CPS) [18, 19, 20, 21, 22, 23, 24, 25]. To tackle the challenge of coordinating the complexity and diversity of tasks, the authors in [18] proposed a task offloading scheme for mobile edge computing (MEC) systems, minimizing deadline violation by using task migration, merging, and deep learning to optimize task execution order and reliability. To tackle the difficulty of agile troubleshooting and deployment of multi-source domain faults in industrial metaverse, the authors in [19] introduced a low-code intelligent fault diagnosis platform, integrating cloud-based infrastructure and edge-adaptive systems to enhance agile fault diagnosis, demonstrated on a Smart Energy Water (SEW) reduction platform and wind turbine generator. Furthermore, by exploiting the accurate digital representation of heterogeneous end-device and network state parameters in dynamic and complex Industrial Internet of Things (IIoT) scenarios in CPS, DT models can be constructed at the edge to capture time-varying demands with low latency [20]. The authors in [21] proposed a cloud-edge-end collaborative intelligent service computation offloading scheme based on the DT driven Edge Coalition Formation methodology to improve the offloading efficiency and utility of edge servers. efficiency and total utility. On the other hand, to ensure accurate and scalable synchronization of industrial Metaverse, [25] proposed a decentralized architecture that partitions the Metaverse into sub-metaverses hosted at the edge, each synchronized with corresponding digital twins using an optimal transport-based assignment scheme.

2. Cross-System Design for Complex Systems: Significant contributions have been made in the area of cross-system system design by jointly considering the communication, computing, and Control [26, 27, 28, 29]. The authors in [26] introduced an integrated scheduling method for sensing, communication, and control in Unmanned Aerial Vehicle (UAV) networks using mmWave/THz communications, enhancing backhaul data transmission. They analyzed the interaction between sensing and motion control, introducing a “state-to-noise-ratio” concept linking control patterns to data rates. The authors in [27] optimized the data transmission in IIoT systems, which derived the average Age of Information (AoI) expression under packet loss and finite retransmissions, linking it to control performance and communication energy consumption. The authors in [28] considered Ultra-Reliable Low-Latency Communication (URLLC) scenarios, where mobile devices predict future states and send them to a data center. They optimized resources to improve delay-reliability tradeoffs, accounting for prediction errors and packet losses. The authors in [29] introduced a proactive tile-based video streaming method for wireless Virtual Reality (VR) to reduce Motion-To-Photon (MTP) latency by predicting and delivering tiles before playback. However, most of these efforts are not task-oriented, resulting in a gap between traditional metrics for communications and the KPIs. In addition, these efforts do not directly consider the interaction between the human and the environment in industrial Metaverse.

3. Task-Oriented Communications Design for Intelligent Device: Unlike traditional methods that measure success based on bit error rates, task-oriented communication focuses on achieving the final task as the measure of success, which has been growing applied to a wide range of industrial Metaverse including IIoT [30],  Unmanned Ground Vehicle (UGV) and  UAV communication systems [31] and industrial Metaverse [32]. In [30], the authors considered a Wireless Networked Control System (WNCS) scenario, where a goal-oriented scheduling and control co-design policy is proposed to minimize violation probability by prioritizing goal-relevant information under resource constraints. In [31], the authors considered UAV tracking scenarios, where a goal-oriented communication framework using Deep Reinforcement Learning (DRL) is proposed to optimize control and communication data selection and repetition. The authors in [32] developed a task-oriented cross-system design framework to minimize packet rates for accurate robotic arm modeling in the Metaverse. However, the robotic arms in the Metaverse simply act as “Digital Shadows” and do not have the ability to synthesize information from the environment as well as user inputs to make decisions on their own accord [33]. In addition, the system design was based on offline data training and testing and did not directly address the latency problem of real-time systems with a human in the loop. The author in [34] developed a similar framework for the Metaverse, but it was only optimized based on a single task, and did not consider generalizability and robustness across multiple tasks.

I-B Contributions

In this paper, we aim to solve the following problems: 1) How to eliminate the effects of time delay among different subsystem components to achieve real-time interactions? 2) How to design a strategy to leverage user behavior to address real-time HITL interactions, while ensuring the algorithm’s generalizability and robustness across various robotic arm operating behaviors? 3) How to build the prototype to verify the effectiveness of the proposed algorithm to ensure a reliable and adaptive industrial Metaverse operation? The main contributions of this paper are summarized as follows:

• •

We propose a task-oriented cross-system design framework for real-time human–robot interaction in industrial Metaverse, tailored for remote inspection and manipulation in nuclear decommissioning. The framework jointly optimizes sensing, communication, prediction, control, and rendering by decoupling the digital twin into proactive visual feedback and predictive control modules.

• •

We develop a behavior-aware horizon management algorithm that dynamically adjusts prediction windows for rendering and control. A two-stage learning process combines offline Model-Agnostic Meta-Learning (MAML)-based pretraining and online HITL adaptation, enhancing robustness in precision tasks such as tile inspection and component dismantling.

• •

We implement a complete industrial Metaverse prototype integrating haptic input, visual feedback, a virtual twin, and a physical robotic arm for a simulated reactor tile inspection task. Experiments under varying communication delays show that our method improves accuracy and responsiveness over baseline approaches, demonstrating its effectiveness for real-world nuclear maintenance operations.

The rest of this paper is organized as follows. In Section II, we propose the task-oriented cross-system design framework where all subsystems, i.e., sensing, communication, prediction, control, and rendering, are elaborated in detail. In Section III, we first briefly introduce the background of MAML and introduce our two-stage training algorithm. Section IV consists of a description of our prototype, as well as the data collection process. Section V provides the training settings and performance evaluations. Finally, Section VI concludes this paper.

II Task-Oriented Cross-System Design

In this section, we propose a task-oriented cross-system design framework. We introduce two functions of the framework, different sub-modules, the time flow chart, and the time delay composition.

II-A System Overview

The proposed framework, illustrated in Fig. 1, is designed to enable real-time human–robot interaction for remote inspection and manipulation tasks in nuclear decommissioning environments. It comprises three spatially distributed components: 1) an operator interface equipped with a haptic input device and a visual feedback display; 2) a virtual environment hosted on an edge server, containing a DT of the robotic arm and a simulated workspace; and 3) a physical environment located within a nuclear fusion reactor vessel, where a real robotic arm operates to carry out inspection and dismantling tasks. These components communicate via a wireless network, simulating the constrained and high-latency communication conditions typically encountered in nuclear fusion facilities. During operation, the human operator observes the DT and its surrounding environment via the visual interface and issues control commands using the input device. The DT module integrates the operator’s intent, virtual environment feedback (e.g., simulated collisions or geometry), and real-time data from the physical robot to generate predictive control instructions. These commands are transmitted to the robotic arm deployed inside the reactor vessel, where it performs precision tasks such as in-vessel tile inspection, alignment, or removal. The resulting physical interactions are sensed, transmitted back to the DT, and displayed to the operator, forming a closed-loop control system.

However, due to time delays introduced by the subsystems including sampling, computation, control, communication and rendering, discrepancies arise between the perceived trajectories of the DT of the robotic arm, the real-world robotic arm, and the operator’s input commands. Even worse, in this HITL industrial Metaverse, time delays in visual feedback can lead to delays in the operator’s control, and delays in control can further affect the generation of visual feedback, which finally significantly affects the robotic arm’s performance of the task. To tackle this problem, the system predicts the operator’s commands and dynamically changes the prediction horizons in real time. This prediction serves two main purposes: (1) to compensate for the delay between the operator’s motion and the visual feedback rendering of the DT, and (2) to minimize the latency between the operator’s control inputs and the actions of the real-world robotic arm in the real-world environment.

II-B Information Flow

As shown in Fig. 2, time is divided into time slots. We modified the entire working process queue as $M/M/1/2^{*}$ [35]. In this case, only one packet is kept in the buffer waiting for processing, and packets waiting for processing are replaced upon the arrival of a more up-to-date packet. No packets are blocked from entering the queue. If a new packet arrives while the system is full, the packet waiting is discarded. The proposed framework comprises two information flows served for two functions: 1) real-world robotic control and 2) visual feedback.

The function of real-world robotic control, encompasses all processes from sensing the movement of the operator through data transmission, the Metaverse data processing, and real-world robotic control. During the process, the operator operates the input device and generates a series of poses $\{{\bf{p}}_{\text{1}},{\bf{p}}_{2},...,{\bf{p}}_{i}\}$ which are sampled by the built-in sensor. Next, In the $t_{1}$-th time slot, an agent decides the length of the prediction horizon. followed by a predictor generates the predicted pose for the visual feedback $\hat{{\bf{p}}}_{v}(t_{1})$ and real-world robotic control $\hat{\bf{p}}_{r}(t_{1})$. Then, $\hat{{\bf{p}}}_{v}(t_{1})$ and $\hat{{\bf{p}}}_{r}(t_{1})$ are transmitted to industrial Metaverse via networks. After that, in the $t_{2}$-th time slot, with the feedback of the environment, the received predicted poses $\hat{{\bf{p}}}_{r}(t_{2})$ is processed to generate a control command in industrial Metaverse, and the Metaverse updates based on the corresponding command and the feedback of the environment. In the $t_{3}$-th time slot, to maintain smoothing motion trajectories in real-world workspace, the command is further interpolated to generate a joint position control command and transmitted via a communication channel for actuation. Finally, in the $t_{8}$-th time slot, the command is received by the real-world robotic arm and further transformed into low-level commands that can be directly executed. Considering the completed real-world robotic control workflow, we define the End-to-End (E2E) delay for the real-world robotic control by:

[TABLE]

On the other hand, the function of the visual feedback, encompasses all processes from the sensing of movement of the operator, data transmission, data processing, rendering, and feedback display. Specifically, we repeat the steps before the process of those in the robotic control function. In addition, in the $t_{4}$-th time slot, the received predicted poses $\hat{{\bf{p}}}_{v}(t_{4})$ is processed to generate a control command, After that, in the $t_{5}$-th time slot, the update in the Metaverse is simulated based on the corresponding command and the feedback of the environment. Next, in the $t_{6}$-th time slot, the virtual environment is rendered. Finally, in the $t_{7}$-th time slot, the rendered image ${\textbf{I}}_{v}(t)$ is streamed back to the user side. Similarly, we define the E2E delay for the visual feedback by:

[TABLE]

II-C System Components

II-C1 Operator Side

On the operator side, based on the position of the robot arm in the visual display, the operator controls the input devices to issue control commands $\mathbf{p}(t)=[l_{\mathrm{x}}(t),l_{\mathrm{y}}(t),l_{\mathrm{z}}(t),q_{\mathrm{x}}(t),q_{\mathrm{y}}(t),q_{\mathrm{z}}(t),q_{\mathrm{w}}(t)]$, where $[l_{\mathrm{x}}(t),l_{\mathrm{y}}(t),l_{\mathrm{z}}(t)]$ is the desired position of in the Cartesian coordinate system and $[q_{\mathrm{x}}(t),q_{\mathrm{y}}(t),q_{\mathrm{z}}(t),q_{\mathrm{w}}(t)]$ is the desired orientation in quaternion [36]. To compensate the time delay latent in all the subsystems, we introduce the predictor to predict the operator’s pose which is used to 1) proactive rendering of virtual environments and 2) preemptive generation of control commands for the real robotic arm. Specifically, in the $t_{1}$-th time slot, given historical poses $\mathcal{P}(t_{1})=[{\bf{p}}(t_{1}-W_{p}),{\bf{p}}(t_{1}-W_{p}+1),...,{\bf{p}}(t_{1})]$, the predictor predicts the pose for the real-world robotic arm control $\hat{\bf{p}}_{\it{r}}(t_{1})$, and the proactive rendering $\hat{\bf{p}}_{\it{v}}(t_{1})$, respectively. This is expressed by

[TABLE]

where $\theta_{p}$ denotes the parameters of the prediction. $H_{c}(t_{1})$ and $H_{r}(t_{1})$ represent the lengths of prediction horizons. To illustrate, we propose to use Autoregressive Moving Average (ARMA) for its low computational complexity, robustness, and no need for pre-training [37], but the framework works equally well with other predictors. Specifically, in the $t_{1}$-th time slot, The one-step predicted value $\hat{\mathbf{p}}(t_{1})$ is expressed as a combination of Auto-Regressive and Moving-Average components, formulated as follows:

[TABLE]

where $\hat{\mathbf{p}}(t_{1})$ is the predicted value at time $t_{1}$, $c$ is the constant term that allows the model to fit time series data with a non-zero mean, $\phi_{a}$ and $\theta_{b}$ are the coefficients of the Auto-Regressive and Moving-Average components, respectively, and $\epsilon(t_{1}-b)$ denotes the prediction errors. The Auto-Regressive term $\sum_{a=1}^{p}\phi_{a}\mathbf{p}(t_{1}-a)$ captures the influence of the $p$-lagged past values of $\mathbf{p}(t_{1})$, while the Moving-Average term $\sum_{b=1}^{q}\theta_{b}\epsilon(t_{1}-b)$ models the effects of past forecast errors up to a lag of $q$. By recursively applying the ARMA model over $H$ steps, we create a sequence of future predictions based on the current and previous observations.

In addition, to dynamically capture the time delay characters in different subsystems, we introduce a DRL agent that adaptively decides the prediction length $H_{r}(t_{1})$ and $H_{v}(t_{1})$, which is expressed by

[TABLE]

where $T_{r}(t_{1}),T_{v}(t_{1})$ are the measured E2E delay defined by $T_{r}=t_{9}-t_{1}$ and $T_{v}=t_{10}-t_{1}$.

II-C2 The Metaverse

The virtual environment, deployed on an edge server, generates 1)control commands for the real-world robotic arm and 2)visual feedback for the operator. At time slot $t_{2}$, the DT synthesizes operator-predicted commands $[\hat{{\bf{p}}}_{r}(t_{1}),\hat{{\bf{p}}}_{v}(t_{1})]$, the last simulation state $\sigma_{v}(t_{2})$, and real-world feedback $\sigma_{r}(t_{2})$ to compute control instructions:

[TABLE]

where $\theta_{c}$ represents control parameters, and $\widetilde{\tau}_{i}(t_{2})$ is the $i$-th joint angle at $t_{2}$.

To ensure smooth motion, we employ Riemannian Motion Policies (RMPFlow), a method merging local strategies into a global policy [38]. The desired acceleration for each joint is:

[TABLE]

where ${\bf{q}}_{i}(t_{2})$ is the current joint angle, ${\bf{q}}^{\prime}_{i}(t_{2})$ the target position, $k_{p}$ the position gain, and $k_{d}$ the damping gain. The robust capping function $\mathcal{R}(\cdot)$ prevents excessive corrections:

[TABLE]

Since real-world execution requires control adaptation, we apply trajectory smoothing via linear interpolation [39]:

[TABLE]

Here, $T_{co}$ is the interval between interpolated commands, and $\alpha_{s}$ evolves as $\alpha_{s}^{\prime}=\alpha_{s}^{2}$.

For visual feedback, the DT updates its virtual model at $t_{4}$ using operator commands $\hat{{\bf{p}}}_{r}(t_{4})$, the simulation state $\sigma_{v}(t_{4})$, and real-world feedback $\sigma_{r}(t_{4})$:

[TABLE]

where $\theta_{v}$ governs visual feedback functions. Simulation updates, including object interactions and collision detection, occur every $\Delta_{t}$ time steps:

[TABLE]

where $\theta_{u}$ defines physics-based updates, and $\theta_{e}$ models robot-environment interactions.

At $t_{6}$, the edge server renders the updated simulation:

[TABLE]

where $\theta_{re}$ determines rendering parameters. The frame rate is given by $f_{r}=\frac{1}{T_{im}}$, where $T_{im}$ is the rendering time per frame. The final image ${\bf{I}}_{v}(t_{6})$ is transmitted to the operator and displayed at $t_{10}$.

II-C3 Real-World Workspace

In real-world workspace, the received motion control commands are further transformed into low-level commands that can be directly executed by the hardware by Application Programming Interfaces (APIs). In the $t_{7}$-th time slot, the joint trajectory of the real-world robotic arm to be executed is expressed by

[TABLE]

where $\mathcal{F}_{rw}(\cdot,\theta_{rw})$ represents the API, and $\theta_{rw}$ denotes the corresponding parameters.

To illustrate, we propose to use the Proportional Integral Derivative (PID) control algorithm for its common ways to implement the control API. Specifically, for the $i$-th joint of a robot, the joint position command in the $t_{7}$-th time slot is calculated by

[TABLE]

where $K_{p},K_{i}$ and $K_{d}$, all non-negative, denote the coefficients for the proportional, integral, and derivative terms, respectively, $\dot{t}$ represents the differentiation of $t$. The control is finished in the $t_{9}$-th time slot. Various sensors are also deployed in the real workspace to monitor the completion of the real-world robotic arm’s tasks and feed the results back to the Metaverse on the edge server.

II-D Task Success Measurements

To evaluate the effectiveness of the design, we conduct a rigorous assessment based on two dimensions: Trajectory Fidelity and Task Accomplishment. To this end, we have designed two types of remote interaction tasks:

• •

Trajectory Tracking Task: In this task, the operator is required to follow a predefined trajectory. The task performance is directly linked to the accuracy of the trajectory completion—the better the trajectory is followed, the higher the task performance. This task aims to assess the system’s real-time control capabilities and path accuracy, simulating remote operations such as surface following and contour scanning.

• •

Open-Ended Task: Unlike the trajectory-based task, this task does not rely on specific paths. The operator is given a fixed amount of time to perform any operation, with the task’s success determined by the quality of the final result. This task evaluates the system’s responsiveness and its ability to handle more flexible, unstructured tasks within a time constraint, reflecting real-world scenarios where operations may vary in complexity.

These two tasks allow for a comprehensive evaluation of the system, from both the precision in task execution (trajectory tracking) and the quality of task outcomes (open-ended operation), ultimately validating the effectiveness of the remote interaction design under different operational conditions. Here is the detail:

II-D1 Trajectory Tracking Tasks-Drawing Shapes

In this task, the human operator manipulates the robotic arm via a haptic input device to trace predefined geometric patterns such as circles, triangles, stars, and squares. These trajectories emulate typical operations in remote handling, including precision surface following and contour scanning. The primary objective is to minimize the positional and orientational discrepancies between the intended trajectory and the executed path of the robotic arm, thereby reflecting the system’s real-time responsiveness and motion fidelity. To achieve accurate alignment between the operator’s input and the robot’s execution, the input data must be transformed between different coordinate spaces. Since the input device and robotic arm operate in different workspaces, input values are mapped to the arm’s coordinate system. The end effector’s pose, represented in Cartesian coordinates and quaternions, is transformed using a scaling matrix $s$, rotation matrix $R_{p}$, and translation matrix $d$:

[TABLE]

For orientation, quaternions are converted into a rotation matrix $R_{i}$, adjusted by a predefined rotation matrix $R_{o}$, and converted back to quaternions.

The error due to MTP latency is quantified as a weighted Root Mean Squared Error (RMSE) between operator motion $\tilde{p}_{i}$ and the rendered virtual arm pose $\tilde{p}_{v}$:

[TABLE]

Similarly, the real-world execution error is computed as:

[TABLE]

These metrics reflect the system’s ability to maintain high-fidelity trajectory following despite the presence of communication delays, mapping inaccuracies, and interface limitations, thereby serving as task-specific KPIs for real-time control quality.

II-D2 Open Tasks: Real-time 3D Scene Representations for Remote Inspection

In nuclear fusion decommissioning, remote inspection is essential due to radiation, contamination, and structural degradation, making manual inspection infeasible. Robotic systems are used to gather visual and geometric data from hazardous areas. Accurate 3D scene representations are crucial for damage assessment and planning tasks like remote cutting and welding. However, challenges such as poor lighting, reflective surfaces, and limited camera viewpoints complicate the generation of high-fidelity models. To address this, an open-ended task is designed where the operator guides a robotic arm to capture images from multiple perspectives within a fixed time. These images are processed using a NeRF-based algorithm to create a 3D model. The task evaluates the system’s performance in latency-sensitive inspection scenarios, with model quality assessed using visual metrics like Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS). The goal is to assess how well the system maintains inspection fidelity under real-time constraints in high-risk environments. Specifically, let $\{{\bf{I}}_{v}(t_{i})\}$ be the set of images captured at viewpoints, where $t_{i}\in[0,T_{task}]$ represents the time at which each image is captured. The task is to generate a 3D model $\mathcal{M}$ from these images:

[TABLE]

Thus the quality of the model is evaluated using the KPIs:

[TABLE]

These KPIs collectively reflect the system’s ability to deliver high-fidelity 3D scene representations under time and communication constraints, which is critical for effective and safe decision-making in industrial real-time Human-robot interaction.

III Human-in-the-loop Model-Agnostic Meta-Learning

In this section, we introduce the HITL-MAML algorithm, where a two-stage training process is included. In the first stage, we train the MAML based on the collected input data from the operators, functioning as a preheater that converges on multiple pre-defined tasks [40]. In the second stage, we use a human-in-the-loop online training strategy, where the operator interacts with industrial Metaverse in real-time, controlling the input device to generate control commands, and receive visual feedback to complete pre-defined tasks. In this stage, the model converges further based on the user’s personalised characters and behaviours.

III-1 State

The state in the $t$-th time slot is a combined vector of the latest min-max normalized input pose ${\bf{p}}(t)$ and the E2E latencies for real-world robotic control and the visual feedback, denoted by ${\bf{s}}_{t}=[{\bf{s}}_{t}^{[1]},{\bf{s}}_{t}^{[2]}]=[{\bf{p}}(t),T_{r}(t),T_{v}(t)],\ {\bf{s}}^{[1]}_{t}={\bf{p}}(t)\in[-1,1]^{1\times 7},\ {{\bf{s}}^{[2]}_{t}}=[T_{r}(t),T_{v}(t)]\in\mathbb{N}^{1\times 2}$.

III-2 Action

In the $t$-th time slot, the action to be taken are the lengths of prediction horizons for real-world robotic arm control $H_{r}(t)$ and for visual feedback $H_{v}(t)$, denoted by ${{\bf{a}}_{t}}=[{{\bf{a}}^{[1]}_{t}},{{\bf{a}}^{[2]}_{t}}]=[H_{r}(t),H_{v}(t)]$.

III-3 Instantaneous Reward

In the $t$-th time slot, given state ${\bf{s}}_{t}$ and action ${\bf{a}}_{t}$, the instantaneous reward is a weighted sum of RMSEs, denoted by

[TABLE]

where ${\bf{e}}_{r}(t)$ is the RMSE for the real-world robotic control and ${\bf{e}}_{v}(t)$ is the RMSE for the visual feedback.

III-4 Policy

The policy $\pi_{\theta}(\mathbf{a}_{t}\mid\mathbf{s}_{t})$ maps states to action probabilities with parameters $\theta$. It consists of multiple fully connected layers: two initial layers extract compact features from input $[\mathbf{p}(t),T_{r}(t),T_{v}(t)]$, followed by layers optimizing prediction horizons for real-world and visual controls. A final linear layer with Softmax generates distributions $\rho_{t}^{[1]}\in\mathbb{R}^{H_{r}}$, $\rho_{t}^{[2]}\in\mathbb{R}^{H_{v}}$. Actions $\mathbf{a}_{t}^{[i]}$ are sampled as:

[TABLE]

where $H$ is the maximum prediction length.

III-A Preliminary of MAML

MAML is a widely used algorithm in meta-learning, where the goal is to quickly adapt a model to new tasks with limited data [40]. The idea is that the convergence of the model on multiple tasks can be transformed into an aggregation that converges on each task. In our algorithm design, we use an extended version of its reinforcement learning to make it generalizable across multiple predefined real-time interactive tasks, as well as allowing for rapid adapting to the second stage online learning. During the process, the model is trained to be able to adapt to a large or infinite number of tasks. Here, we denote that there are $N\in\mathbb{N}^{+}$ tasks, ${\bf{T}}={T_{1},T_{2},\dots,T_{N}}$, and each task has finite length $L$. For example, The $n$-th task $T_{n}$ can be defined as

[TABLE]

where $\{({\bf{a}}_{t_{1}},{\bf{s}}_{t_{1}}),...,({\bf{a}}_{t_{L}},{\bf{s}}_{t_{L}})\}$ represents a sequence of $L$ state-action pairs generated by following policy $\pi_{\theta}$, $\rho_{t}$ is the transition distribution of actions.

Here, we adapt the loss function in MAML by incorporating the Proximal Policy Optimization (PPO) loss function [41],

[TABLE]

where $\theta^{k}$ represents the parameters of policy network in previous $k$ steps, $A({\bf{s}}_{t},{\bf{a}}_{t})$ is the advantage function which estimates the advantage of taking action ${\bf{a}}_{t}$ in state ${\bf{s}}_{t}$, over other possible actions in the same state [41],

[TABLE]

where $Q^{\pi_{\theta}}({\bf{s}}_{t},{\bf{a}}_{t})$ is the state-action value function and $V^{\pi_{\theta}}({\bf{s}}_{t})$,is the state-value function. Then, based on (23), we compute the adapted parameters with one step gradient descent by each time sampling $K$ trajectories under policy $\pi_{\theta_{n}}$ for task $T_{n}$,

[TABLE]

where $\alpha$ represents the step length and $\nabla_{\theta}$ is the gradient with respect to the parameter ${\theta}$. After performing (25) for each task ${\bf{T}}={T_{1},T_{2},\dots,T_{N}}$, we then implement the meta-optimization via Stochastic Gradient Descent (SGD), where the $\theta$ are further updated as follows,

[TABLE]

where $\beta$ represents the meta-step length.

III-B Problem Formulation

We optimize the two prediction horizons for functions 1) real-world robotic arm control, and 2) visual feedback, to minimize the weighted sum of 1) the RMSE between the operator’s input pose and the end-effector of the robotic arm $e_{r}(t)$, and 2) the RMSE between the operator’s input pose and the pose of the DT of the robotic arm displayed on the screen $e_{v}(t)$ in $N\in\mathbb{N}^{+}$ tasks, ${\bf{T}}={T_{1},T_{2},\dots,T_{N}}$. For this purpose, we formulate it as a meta-reinforcement learning problem to find the optimal policy $\pi_{\theta}^{*}$,

[TABLE]

where $\gamma$ represents the discount factor, $H_{v}$ denotes the maximum prediction horizon for the rendering loop and $H_{r}$ is the maximum prediction horizon for the control loop.

III-C Two-Stage Training Process

Our proposed two-stage training algorithm is shown in Algorithm 1. In the first stage, we train the  MAML based on the collected data which is detailed in Section IV-B. The goal of this stage serves as a warm-up phase, which enables the agent to quickly acquire a policy for the next stage of training, i.e., real-time interaction using only a small amount of experience. In the second stage, HITL online training is shown in Fig. 3. The operator interacts with a reinforcement learning environment - manipulating an input device in real time based on visual feedback to control a real-world robotic arm to perform a new interaction tasks. In this process, states and actions are generated online and reward is computed in real time. As the interaction continues, the model is trained until it converges.

IV Prototype Design

IV-A System Setup

IV-A1 Input Device

As shown in Fig. 4, the Touch Haptic Device is employed as the primary input mechanism, enabling the operator to seamlessly interact with the system [42]. This device provides six degrees of freedom (DoF) for position and orientation sensing, alongside three degrees of force feedback. These capabilities offer an intuitive and immersive control interface, allowing for a tactile interaction experience. With a positional accuracy of up to 0.055 mm and a force feedback resolution as fine as 0.03 N, the device ensures precise input tracking, making it well-suited for tasks that require meticulous control. This high level of accuracy facilitates the fine manipulation of virtual objects, contributing to a smooth and responsive interaction between the operator and the Metaverse. In the context of our framework, the Touch Haptic Device enhances the operator’s ability to perform high-precision tasks with minimal error. Its integration into the system is crucial for scenarios demanding delicate and accurate manipulation, thus improving the overall functionality and efficiency of the framework.

IV-A2 Visual Feedback

We utilize a Dell G3223Q 4K monitor displaying the virtual work environment hosted on the edge server [43]. This monitor provides high-resolution visual feedback of the DT of the robotic arm and the Metaverse, with a refresh rate of 165 Hz, ensuring smooth and uninterrupted viewing. This allows the operator to assess the extent of the task based on visual feedback and make timely control via the touch device. Thus, the combination of control and visual feedback enhances the overall interaction, ensuring optimal performance in the proposed industrial Metaverse.

IV-A3 The Metaverse

The Metaverse is built on the NVIDIA Omniverse platform, utilizing Isaac Sim to simulate and control robotic systems [44], which is running on a high-performance edge server equipped with 32 GB RAM, an NVIDIA RTX 3070 GPU, and an Intel Core i7-11700 16-core CPU. We employ Isaac Sim to create a highly accurate DT of the real-world robotic arm, modeling its dimensions and functionalities in detail. In addition to the physical simulation, our framework integrates Isaac Cortex, which enhances task awareness and allows for more advanced decision-making capabilities in robotic systems [45]. Instead of merely simulating physical interactions, our framework uses this decision-making layer to optimize how robots approach and complete tasks, leading to more efficient operations. We also decouple the physics updates from visual rendering in Isaac Sim, allowing us to maintain high-frequency physical updates while ensuring smooth visual feedback on the operator’s monitor. This setup ensures that the operator receives continuous real-time feedback while controlling the virtual environment, ensuring accurate task execution. The overall framework, with its integration of Isaac Sim and Isaac Cortex, creates a tightly coupled system between the operator side, the Metaverse, and the real-world robotic arm, enabling precise and efficient task management.

IV-A4 Real-World Robotic Arm

In our framework, we employ the Universal Robots UR3e industrial robotic arm as the core component for executing physical tasks [46]. The UR3e offers six DoF and achieves a repeatability of ±0.03 mm, making it well-suited for tasks that require high precision. With a payload capacity of 3 kg and a reach of 500 mm, the UR3e is optimized for small-scale, detailed operations while maintaining flexibility in diverse working environments. In the real-world workspace, the UR3e interacts with objects that are digitally mirrored in the Metaverse, ensuring seamless integration between the cyber and physical domains. After receiving the control command from the edge server, the UR3e implements the control commands by utilizing the MoveIt Kinematics Plugin [47], which enables smooth and accurate joint motion execution. In addition, during the entire working process, the built-in sensors of the robotic arm monitor the joint angles, angular velocities, and applied forces to maintain consistent and precise control. This sensing system is critical for maintaining the accuracy required in our task-oriented framework, ensuring that every command is executed with the necessary precision and accurate task-oriented KPIs calculations.

IV-A5 Networks

Our framework implements a publish-and-subscribe mechanism in the Robot Operating System (ROS) environment to facilitate communication between the operator, the edge server, and the real-world workspace [48]. This mechanism allows for flexible and scalable data exchange, enabling the modular development of complex cyber-physical applications. Communication between processes relies on two protocols: Remote Procedure Call (RPC) and Transmission Control Protocol (TCP) [49]. When transmitting data, the local process first involves a publisher (talker) registering with the ROS Master on a designated port, including the topic name in its registration. A subscriber (listener) then registers with ROS Master and requests information based on its subscription. The ROS Master, upon finding a matching topic publisher, provides the listener with the talker’s RPC address. The listener sends an RPC request to the talker, initiating the connection by specifying the topic name and message type. Once the talker confirms this request, it provides its TCP address, and the listener establishes a network connection. The talker can then begin publishing data to the listener over TCP. To simulate and control communication delays within this network, we used the network cables to connect the three via Ethernet during training, making the communication delay of the benchmarks nearly zero. Then, we utilize the Traffic Control (TC) method in Linux [50]. This utility allows us to manipulate network delay by introducing artificial delays, enabling controlled experiments on how varying levels of network delay affect the system’s performance. By systematically varying communication delays with TC, we can observe their impact on the control and response of the robotic system. These controlled network conditions allow us to generate reproducible results for training and testing.

IV-B Data Collection

As shown in fig. 5, we collected real-time interaction data from four trajectory-following tasks, where an operator used a touch haptic device to control a robotic arm through shapes including circle, triangle, pentagram, and square. These trajectories are designed to simulate typical motion patterns in nuclear decommissioning tasks, such as remote cutting, surface following, and precise manipulation of in-vessel components [51]. The operator received real-time 3D visual feedback from a digital twin, enabling on-the-fly adjustments to trajectory parameters (e.g., shape, size, and speed) to adapt to dynamic task requirements and workspace constraints. Both the target pose from the haptic device and the actual pose of the robot end-effector were continuously recorded in Comma Separated Values (CSV) format. Data collection consisted of two phases: (1) 150 repetitions of each trajectory shape for pretraining, simulating repetitive precision operations, and (2) online learning during novel, task-specific executions until model convergence. This dataset forms the foundation for training and validating our predictive control framework under realistic remote operation conditions relevant to nuclear inspection and maintenance.

IV-C Validation in Remote Inspection Tasks for Nuclear Decommissioning

In addition to implementing HITL-MAML online training on predefined trajectories, we validate the effectiveness of HITL-MAML in a real-world industrial application focused on high-precision 3D visualization and inspection of reactor tiles in nuclear decommissioning. For evaluation, we use full-scale 3D-printed replicas of reactor tiles. Expert operators remotely control the robotic arm along predefined paths, pausing at strategic points using the haptic touch interface to capture images. The 3D scene representation server generates 3D models from these images and camera poses using a NeRF-based algorithm, enabling realistic remote inspection with novel view synthesis [52, 53]. However, due to the communication delays inherent in remote control, operators experience noticeable lag, which can hinder their ability to control the robotic arm precisely within a limited timeframe. This time lag can affect the accuracy of the arm’s movements and the overall inspection process [54]. To address these challenges, we apply HITL-MAML, assessing its effectiveness in improving control precision and 3D inspection accuracy under latency constraints. Experimental details will be provided in the evaluation section.

V Results

V-A Training Settings

Algorithm training was conducted on a server with an Intel Core i9-13900F CPU, 64 GB RAM, and an NVIDIA RTX 4090 GPU. The default parameters of the prototype and learning algorithm are listed in Table I, unless otherwise specified. The parameters of HITL-MAML are updated following the procedure outlined in Algorithm 1. In the first stage, we use the collected motion data mentioned in Section IV-B. The framework sequentially executes each drawing shape over $4\times 10^{4}$ ms. In the second stage, the operator interacts with the DRL environment: 1) HITL training for incorporating human operation;2) 3D scene representations for remote inspection. The training time of each drawing shape is set to $2\times 10^{4}$ ms. To simulate network delays $T_{d}$, a normal distribution with a mean of $\mu_{c}=50$ ms and a standard deviation of $\sigma_{c}=10$ ms is set between the operator side and the edge server.

V-B Performance Evaluation of the Prediction Model

As shown in Fig. 6, the performance of the predictor ARMA is evaluated by the RMSE with different prediction horizons. The results show that the predictor achieves similar performance in four predefined trajectories, with less than 0.01m RMSE when the prediction horizon is less than 1000 ms. It is intuitively that as the prediction horizon increases, the prediction error also rises. Thus, there exists an optimal point where there is neither an increase in error due to prediction by compensating for too much delay, nor an insufficient prediction where there is still an error. Establishing this level of prediction accuracy is crucial, as it enables robust performance when dynamically adjusting prediction horizons during interactions with the DRL environment.

V-C Performance Evaluation of Trajectory Tracking Tasks-Drawing Shapes

V-C1 Performance on the Collected Dataset

As shown in Fig. 7, we train the MAML algorithm over 500 episodes for three times to evaluate the proposed MAML training process on the collected dataset. To better illustrate our algorithm in the figure, we denote the MAML training process on the collected dataset as ”MAML-CD”. The RMSE gradually decreases during the first 300 episodes and achieves convergence at an RMSE of $0.0096$ m after 350 episodes of training. “Std” represents the standard deviation. To verify the effectiveness of the proposed algorithm, We compare the performance of the trained MAML with three baselines,

• •

WP: Without any delay compensation strategy, the pose of the input device is directly transmitted.

• •

RS: We predict the pose for the two functions, where the prediction horizons $H_{r}(t)$ and $H_{v}(t)$, are randomly sampled within the maximum value of $H$.

• •

OD: The prediction horizons $H_{r}(t)$ and $H_{v}(t)$, are dynamically changed depends on the measured E2E delays, $T_{r}(t)$ and $T_{v}(t)$. For example, if E2E delay of real-world robotic control and visual feedback is 127ms and 133ms, respectively, we set $H_{r}(t)=127$ and $H_{v}(t)=133$.

The results show a significant superiority over the three baselines. The RMSE decrease by 249.39%, 132.79%, and 61.13% compared to ’WP’, ’OD’, and ’RS’, respectively.

V-C2 Performance of HITL Online Training

As shown in Fig. 8, the performance of the HITL-MAML training process is evaluated by incorporating human operators controlling the input device during training. we compare the results with MAML-CD and the other three baselines. The results show that HITL-MAML rapidly converges after training for 80 episodes with an average RMSE of $0.0101$ m. which is close to the RMSE of MAML-CD. This indicates that our proposed algorithm can quickly adapt to the HITL mode after a short period of online training, which verifies the effectiveness and efficiency of our algorithm. In addition, as shown in Tab. III, we further examined the RMSE of the proposed algorithm in each task, and compared it to the three baselines mentioned above. The results show that the average RMSE of our proposed algorithm is less than that of the three baselines in each task, further illustrating the effectiveness of our algorithm. We get the lowest RMSE in task ’square’, where the average RMSE of position and orientation are $0.0068$ m and $0.0089$, respectively.

V-C3 Evaluation in Different Communication Conditions

As shown in Fig. 10, We evaluate the performance of the proposed algorithm under different communication delay conditions. Specifically, In addition to the [math] ms communication delay as a benchmark, we add different normally distributed delays, with mean $\mu_{c}=50,100$ ms and a standard deviation of $\sigma_{c}=10$ ms is set between the operator side and the edge server. The results show that our proposed algorithm can successfully converge and reach an RMSE of approximately $0.011$ m. We also measured the E2E as shown in Tab. II. The results also indicate that a higher E2E delay needs to make the algorithm take more time to converge.

V-D Performance Evaluation of pen Tasks: Real-time 3D Scene Representations for Remote Inspection

To evaluate the effectiveness of our proposed HITL-MAML algorithm in the context of 3D scene representation for remote nuclear inspection, we designed two task-oriented experimental settings: (1) Fixed Viewpoint Budget: the operator was allowed a fixed number of image captures, and the evaluation focused on task completion time and representation quality; (2) Fixed Time Budget: the operator had a fixed time to explore the environment, and the number of captured viewpoints and representation quality were measured. The quality of Real-time 3D scene representation was assessed both visually (Fig. 9) and quantitatively using PSNR, SSIM, and LPIPS metrics. A professional operator conducted the image-capturing task by remotely controlling the robotic arm using our HITL-MAML framework and three baseline methods. As summarized in Table IV, our method consistently achieved higher representation fidelity and reduced task time, particularly under time-constrained conditions. These results demonstrate the practical advantage of our approach for time-sensitive and communication-constrained inspection tasks.

V-E Demonstration of RMSE

As shown in Fig. 11, we demonstrate the RMSE by showing different trajectories for two seconds. The input device’s trajectories are transformed into the robotic arm’s coordinate system. Due to the fact that the input device is the one that issued the command, it is logical that it is ahead of the other trajectories at the start point. It can be visualized that our proposed algorithm achieves the lowest RMSE at the endpoint, with the point that is closest to that of the input device. This provides a more intuitive illustration of the algorithm’s effectiveness.

VI Conclusions

In this study, we proposed a task-oriented cross-system framework leveraging DTs to facilitate real-time human-device interaction in industrial Metaverse. By decoupling DTs into visual display and robotic control components, the system optimized responsiveness and adaptability under strict computational and network constraints. The integration of the HITL-MAML algorithm enabled dynamic adjustment of prediction horizons, improving the generalizability of operator motion prediction. Experimental evaluations demonstrated the effectiveness of the proposed approach: in the Trajectory-Based Drawing Control task, the framework significantly reduced the weighted RMSE from 0.0712 m to 0.0101 m; in the real-time 3D scene representation task for nuclear decommissioning, it achieved a PSNR of 22.11, SSIM of 0.8729, and LPIPS of 0.1298. These results validated the framework’s capability to ensure both spatial precision and visual fidelity in real-time, high-risk industrial scenarios.