Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL

TL;DR

This paper demonstrates that using simulation to learn exploratory policies can significantly improve real-world reinforcement learning, especially when direct sim2real transfer fails, by providing polynomial sample complexity in low-rank MDPs.

Contribution

It introduces a method to leverage simulation for learning exploratory policies that enable efficient real-world exploration, with theoretical guarantees and practical validation.

Findings

Exploratory policies enable polynomial sample complexity in low-rank MDPs.

Simulation transfer improves real-world RL performance over direct sim2real transfer.

Validated on robotic simulators and real-world tasks, showing practical benefits.

Abstract

In order to mitigate the sample complexity of real-world reinforcement learning, common practice is to first train a policy in a simulator where samples are cheap, and then deploy this policy in the real world, with the hope that it generalizes effectively. Such \emph{direct sim2real} transfer is not guaranteed to succeed, however, and in cases where it fails, it is unclear how to best utilize the simulator. In this work, we show that in many regimes, while direct sim2real transfer may fail, we can utilize the simulator to learn a set of \emph{exploratory} policies which enable efficient exploration in the real world. In particular, in the setting of low-rank MDPs, we show that coupling these exploratory policies with simple, practical approaches -- least-squares regression oracles and naive randomized exploration -- yields a polynomial sample complexity in the real world, an exponential improvement over direct sim2real transfer, or learning without access to a simulator. To the best of our knowledge, this is the first evidence that simulation transfer yields a provable gain in reinforcement learning in settings where direct sim2real transfer fails. We validate our theoretical results on several realistic robotic simulators and a real-world robotic sim2real task, demonstrating that transferring exploratory policies can yield substantial gains in practice as well.