Zero-shot Sim2Real Adaptation Across Environments

TL;DR

This paper introduces a novel Reverse Action Transformation (RAT) policy that enables zero-shot adaptation in sim2real transfer for robotics, reducing the need for environment-specific residual models.

Contribution

The paper proposes RAT, a method that learns to imitate simulated policies in the real world, allowing quick transfer to new environments without retraining residual models.

Findings

RAT outperforms existing methods in continuous control tasks.

RAT enables zero-shot adaptation to new environments.

The approach requires only a single environment to learn effective transfer.

Abstract

Simulation based learning often provides a cost-efficient recourse to reinforcement learning applications in robotics. However, simulators are generally incapable of accurately replicating real-world dynamics, and thus bridging the sim2real gap is an important problem in simulation based learning. Current solutions to bridge the sim2real gap involve hybrid simulators that are augmented with neural residual models. Unfortunately, they require a separate residual model for each individual environment configuration (i.e., a fixed setting of environment variables such as mass, friction etc.), and thus are not transferable to new environments quickly. To address this issue, we propose a Reverse Action Transformation (RAT) policy which learns to imitate simulated policies in the real-world. Once learnt from a single environment, RAT can then be deployed on top of a Universal Policy Network to achieve zero-shot adaptation to new environments. We empirically evaluate our approach in a set of continuous control tasks and observe its advantage as a few-shot and zero-shot learner over competing baselines.