Abstract Sim2Real through Approximate Information States

TL;DR

This paper formalizes the abstract sim2real problem in reinforcement learning, proposing a method to adapt coarse simulators using real-world data for effective policy transfer in robotics.

Contribution

It introduces a formal framework for abstract simulators in RL and a method to ground these models with real-world data for improved sim2real transfer.

Findings

The formalism shows that grounded abstract dynamics must consider state history.

The proposed method successfully adapts abstract simulators using real-world data.

Experiments demonstrate effective policy transfer in both sim2sim and sim2real settings.

Abstract

In recent years, reinforcement learning (RL) has shown remarkable success in robotics when a fast and accurate simulator is available for a given task. When using RL and simulation, more simulator realism is generally beneficial but becomes harder to obtain as robots are deployed in increasingly complex and widescale domains. In such settings, simulators will likely fail to model all relevant details of a given target task and this observation motivates the study of sim2real with simulators that leave out key task details. In this paper, we formalize and study the abstract sim2real problem: given an abstract simulator that models a target task at a coarse level of abstraction, how can we train a policy with RL in the abstract simulator and successfully transfer it to the real-world? Our first contribution is to formalize this problem using the language of state abstraction from the RL literature. This framing shows that an abstract simulator can be grounded to match the target task if the grounded abstract dynamics take the history of states into account. Based on the formalism, we then introduce a method that uses real-world task data to correct the dynamics of the abstract simulator. We then show that this method enables successful policy transfer both in sim2sim and sim2real evaluation.