Understanding Domain Randomization for Sim-to-real Transfer

TL;DR

This paper provides a theoretical framework for understanding why domain randomization effectively enables sim-to-real transfer in reinforcement learning, offering bounds on the transfer gap and emphasizing the role of memory in policies.

Contribution

It introduces a formal model of sim-to-real transfer with bounds on the transfer gap and demonstrates conditions for success without real-world data, highlighting the importance of history-dependent policies.

Findings

Bounds on the sim-to-real gap are established.

Transfer can succeed without real-world training samples.

Memory in policies enhances transfer success.

Abstract

Reinforcement learning encounters many challenges when applied directly in the real world. Sim-to-real transfer is widely used to transfer the knowledge learned from simulation to the real world. Domain randomization -- one of the most popular algorithms for sim-to-real transfer -- has been demonstrated to be effective in various tasks in robotics and autonomous driving. Despite its empirical successes, theoretical understanding on why this simple algorithm works is limited. In this paper, we propose a theoretical framework for sim-to-real transfers, in which the simulator is modeled as a set of MDPs with tunable parameters (corresponding to unknown physical parameters such as friction). We provide sharp bounds on the sim-to-real gap -- the difference between the value of policy returned by domain randomization and the value of an optimal policy for the real world. We prove that sim-to-real transfer can succeed under mild conditions without any real-world training samples. Our theory also highlights the importance of using memory (i.e., history-dependent policies) in domain randomization. Our proof is based on novel techniques that reduce the problem of bounding the sim-to-real gap to the problem of designing efficient learning algorithms for infinite-horizon MDPs, which we believe are of independent interest.