Unlocking the Potential of Simulators: Design with RL in Mind

TL;DR

This paper demonstrates that designing simple, RL-compatible simulators that model control and dynamics can outperform high-fidelity simulators in training policies for real-world robotic tasks, especially when key uncertainties are identified.

Contribution

The authors introduce a novel approach to simulator design that emphasizes control modeling and show its effectiveness in robotic policy learning, challenging the reliance on high-fidelity simulators.

Findings

Simple RL-compatible simulators outperform high-fidelity ones in policy transfer.

Modeling control and key uncertainties enables effective policy learning.

Exploiting phenomena like friction can improve real-world policy performance.

Abstract

Using Reinforcement Learning (RL) in simulation to construct policies useful in real life is challenging. This is often attributed to the sequential decision making aspect: inaccuracies in simulation accumulate over multiple steps, hence the simulated trajectories diverge from what would happen in reality. In our work we show the need to consider another important aspect: the mismatch in simulating control. We bring attention to the need for modeling control as well as dynamics, since oversimplifying assumptions about applying actions of RL policies could make the policies fail on real-world systems. We design a simulator for solving a pivoting task (of interest in Robotics) and demonstrate that even a simple simulator designed with RL in mind outperforms high-fidelity simulators when it comes to learning a policy that is to be deployed on a real robotic system. We show that a phenomenon that is hard to model - friction - could be exploited successfully, even when RL is performed using a simulator with a simple dynamics and noise model. Hence, we demonstrate that as long as the main sources of uncertainty are identified, it could be possible to learn policies applicable to real systems even using a simple simulator. RL-compatible simulators could open the possibilities for applying a wide range of RL algorithms in various fields. This is important, since currently data sparsity in fields like healthcare and education frequently forces researchers and engineers to only consider sample-efficient RL approaches. Successful simulator-aided RL could increase flexibility of experimenting with RL algorithms and help applying RL policies to real-world settings in fields where data is scarce. We believe that lessons learned in Robotics could help other fields design RL-compatible simulators, so we summarize our experience and conclude with suggestions.