Can Context Bridge the Reality Gap? Sim-to-Real Transfer of Context-Aware Policies

TL;DR

This paper explores how conditioning policies on environment dynamics estimates can improve sim-to-real transfer in robotics, demonstrating that context-aware policies outperform traditional methods in benchmarks and real-world tasks.

Contribution

It introduces a framework integrating context estimation into domain randomization-based RL, systematically comparing supervision strategies for improved transfer performance.

Findings

Context-aware policies outperform baseline in benchmarks.

Supervision strategy effectiveness varies by task.

Improved sim-to-real transfer demonstrated on real robot.

Abstract

Sim-to-real transfer remains a major challenge in reinforcement learning (RL) for robotics, as policies trained in simulation often fail to generalize to the real world due to discrepancies in environment dynamics. Domain Randomization (DR) mitigates this issue by exposing the policy to a wide range of randomized dynamics during training, yet leading to a reduction in performance. While standard approaches typically train policies agnostic to these variations, we investigate whether sim-to-real transfer can be improved by conditioning the policy on an estimate of the dynamics parameters -- referred to as context. To this end, we integrate a context estimation module into a DR-based RL framework and systematically compare SOTA supervision strategies. We evaluate the resulting context-aware policies in both a canonical control benchmark and a real-world pushing task using a Franka Emika Panda robot. Results show that context-aware policies outperform the context-agnostic baseline across all settings, although the best supervision strategy depends on the task.