Mixed-Reality Digital Twins: Leveraging the Physical and Virtual Worlds for Hybrid Sim2Real Transition of Multi-Agent Reinforcement Learning Policies

TL;DR

This paper introduces a mixed-reality digital twin framework that accelerates multi-agent reinforcement learning training and enhances sim2real transfer for cyber-physical vehicle systems, reducing training time and sim2real gap.

Contribution

The work presents a novel mixed-reality digital twin framework that improves training efficiency and sim2real transfer in multi-agent reinforcement learning for cyber-physical systems.

Findings

Up to 76.3% reduction in training time with parallelization.

Achieved as low as 2.9% sim2real gap.

Validated on cooperative and competitive MARL use cases.

Abstract

Multi-agent reinforcement learning (MARL) for cyber-physical vehicle systems usually requires a significantly long training time due to their inherent complexity. Furthermore, deploying the trained policies in the real world demands a feature-rich environment along with multiple physical embodied agents, which may not be feasible due to monetary, physical, energy, or safety constraints. This work seeks to address these pain points by presenting a mixed-reality (MR) digital twin (DT) framework capable of: (i) boosting training speeds by selectively scaling parallelized simulation workloads on-demand, and (ii) immersing the MARL policies across hybrid simulation-to-reality (sim2real) experiments. The viability and performance of the proposed framework are highlighted through two representative use cases, which cover cooperative as well as competitive classes of MARL problems. We study the effect of: (i) agent and environment parallelization on training time, and (ii) systematic domain randomization on zero-shot sim2real transfer, across both case studies. Results indicate up to 76.3% reduction in training time with the proposed parallelization scheme and sim2real gap as low as 2.9% using the proposed deployment method.