Trade-off on Sim2Real Learning: Real-world Learning Faster than Simulations

TL;DR

This paper demonstrates that Deep Bayesian Reinforcement Learning outperforms traditional DRL in sample efficiency and real-world learning speed, suggesting that real-world training can be more effective than simulation-based methods for robotic policy learning.

Contribution

The study compares Deep Bayesian RL with model-free DRL, showing Bayesian methods achieve better real-world learning efficiency and highlighting the potential of real-world data to improve policy training.

Findings

Deep Bayesian RL outperforms DRL in sample efficiency.

Real-world learning can be faster than simulation-based training.

Mixing simulation and real-world data does not outperform pure real-world learning.

Abstract

Deep Reinforcement Learning (DRL) experiments are commonly performed in simulated environments due to the tremendous training sample demands from deep neural networks. In contrast, model-based Bayesian Learning allows a robot to learn good policies within a few trials in the real world. Although it takes fewer iterations, Bayesian methods pay a relatively higher computational cost per trial, and the advantage of such methods is strongly tied to dimensionality and noise. In here, we compare a Deep Bayesian Learning algorithm with a model-free DRL algorithm while analyzing our results collected from both simulations and real-world experiments. While considering Sim and Real learning, our experiments show that the sample-efficient Deep Bayesian RL performance is better than DRL even when computation time (as opposed to number of iterations) is taken in consideration. Additionally, the difference in computation time between Deep Bayesian RL performed in simulation and in experiments point to a viable path to traverse the reality gap. We also show that a mix between Sim and Real does not outperform a purely Real approach, pointing to the possibility that reality can provide the best prior knowledge to a Bayesian Learning. Roboticists design and build robots every day, and our results show that a higher learning efficiency in the real-world will shorten the time between design and deployment by skipping simulations.