Cloud-Edge Training Architecture for Sim-to-Real Deep Reinforcement Learning

TL;DR

This paper introduces a cloud-edge architecture for real-time training of deep reinforcement learning agents on physical systems, effectively bridging the reality gap and enabling adaptive control in real-world environments.

Contribution

It proposes a novel distributed cloud-edge framework that separates inference and training, facilitating real-time adaptation of pretrained DRL policies to real-world dynamics.

Findings

Successfully applied to inverted-pendulum control system

Demonstrated effective adaptation to unseen dynamics

Achieved efficient real-time training in physical environment

Abstract

Deep reinforcement learning (DRL) is a promising approach to solve complex control tasks by learning policies through interactions with the environment. However, the training of DRL policies requires large amounts of training experiences, making it impractical to learn the policy directly on physical systems. Sim-to-real approaches leverage simulations to pretrain DRL policies and then deploy them in the real world. Unfortunately, the direct real-world deployment of pretrained policies usually suffers from performance deterioration due to the different dynamics, known as the reality gap. Recent sim-to-real methods, such as domain randomization and domain adaptation, focus on improving the robustness of the pretrained agents. Nevertheless, the simulation-trained policies often need to be tuned with real-world data to reach optimal performance, which is challenging due to the high cost of real-world samples. This work proposes a distributed cloud-edge architecture to train DRL agents in the real world in real-time. In the architecture, the inference and training are assigned to the edge and cloud, separating the real-time control loop from the computationally expensive training loop. To overcome the reality gap, our architecture exploits sim-to-real transfer strategies to continue the training of simulation-pretrained agents on a physical system. We demonstrate its applicability on a physical inverted-pendulum control system, analyzing critical parameters. The real-world experiments show that our architecture can adapt the pretrained DRL agents to unseen dynamics consistently and efficiently.