Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model

TL;DR

This paper presents a method that uses a learned deep inverse dynamics model to transfer control policies from simulation to real robots by predicting suitable real-world actions to achieve simulated next states.

Contribution

The paper introduces a novel approach combining simulation-based policies with deep inverse dynamics models for effective sim-to-real transfer, along with an incremental data collection strategy.

Findings

Outperforms baseline methods like output error control and Gaussian dynamics adaptation.

Effectively predicts real-world actions to match simulated next states.

Enables safer and more practical deployment of policies learned in simulation.

Abstract

Developing control policies in simulation is often more practical and safer than directly running experiments in the real world. This applies to policies obtained from planning and optimization, and even more so to policies obtained from reinforcement learning, which is often very data demanding. However, a policy that succeeds in simulation often doesn't work when deployed on a real robot. Nevertheless, often the overall gist of what the policy does in simulation remains valid in the real world. In this paper we investigate such settings, where the sequence of states traversed in simulation remains reasonable for the real world, even if the details of the controls are not, as could be the case when the key differences lie in detailed friction, contact, mass and geometry properties. During execution, at each time step our approach computes what the simulation-based control policy would do, but then, rather than executing these controls on the real robot, our approach computes what the simulation expects the resulting next state(s) will be, and then relies on a learned deep inverse dynamics model to decide which real-world action is most suitable to achieve those next states. Deep models are only as good as their training data, and we also propose an approach for data collection to (incrementally) learn the deep inverse dynamics model. Our experiments shows our approach compares favorably with various baselines that have been developed for dealing with simulation to real world model discrepancy, including output error control and Gaussian dynamics adaptation.