Learning more with the same effort: how randomization improves the robustness of a robotic deep reinforcement learning agent

TL;DR

This paper demonstrates that introducing randomization during simulation training enhances the robustness of robotic deep reinforcement learning agents, reducing real-world data needs and improving transferability from virtual to real environments.

Contribution

It shows that adding diversity through randomization improves robustness and efficiency of sim-to-real transfer in DRL agents, a novel approach for better sim-to-real performance.

Findings

Randomization increases model accuracy by around 25%.

Enhanced robustness reduces the need for real-world data.

Adding real experience remains beneficial regardless of virtual training quality.

Abstract

The industrial application of Deep Reinforcement Learning (DRL) is frequently slowed down because of the inability to generate the experience required to train the models. Collecting data often involves considerable time and economic effort that is unaffordable in most cases. Fortunately, devices like robots can be trained with synthetic experience thanks to virtual environments. With this approach, the sample efficiency problems of artificial agents are mitigated, but another issue arises: the need for efficiently transferring the synthetic experience into the real world (sim-to-real). This paper analyzes the robustness of a state-of-the-art sim-to-real technique known as progressive neural networks (PNNs) and studies how adding diversity to the synthetic experience can complement it. To better understand the drivers that lead to a lack of robustness, the robotic agent is still tested in a virtual environment to ensure total control on the divergence between the simulated and real models. The results show that a PNN-like agent exhibits a substantial decrease in its robustness at the beginning of the real training phase. Randomizing certain variables during simulation-based training significantly mitigates this issue. On average, the increase in the model's accuracy is around 25% when diversity is introduced in the training process. This improvement can be translated into a decrease in the required real experience for the same final robustness performance. Notwithstanding, adding real experience to agents should still be beneficial regardless of the quality of the virtual experience fed into the agent.