Predicting Sim-to-Real Transfer with Probabilistic Dynamics Models

TL;DR

This paper introduces a probabilistic dynamics model-based transfer metric that predicts the real-world performance of RL policies from simulation, reducing the need for costly real-world testing.

Contribution

It presents a novel transfer metric that correlates with real-world policy success and predicts the impact of training choices without extensive real-world trials.

Findings

Transfer metric correlates with real-world performance

Predicts effects of training setup variations

Reduces need for real-world rollouts

Abstract

We propose a method to predict the sim-to-real transfer performance of RL policies. Our transfer metric simplifies the selection of training setups (such as algorithm, hyperparameters, randomizations) and policies in simulation, without the need for extensive and time-consuming real-world rollouts. A probabilistic dynamics model is trained alongside the policy and evaluated on a fixed set of real-world trajectories to obtain the transfer metric. Experiments show that the transfer metric is highly correlated with policy performance in both simulated and real-world robotic environments for complex manipulation tasks. We further show that the transfer metric can predict the effect of training setups on policy transfer performance.