Sim-to-Real Transfer in Deep Reinforcement Learning for Bipedal Locomotion

TL;DR

This paper explores the challenges of transferring deep reinforcement learning policies from simulation to real-world bipedal robots, analyzing the sources of the sim-to-real gap and proposing strategies to improve transfer robustness.

Contribution

It systematically analyzes the sources of the sim-to-real gap in bipedal locomotion and proposes a combined framework of model fidelity improvement and policy hardening techniques.

Findings

Identified key sources of sim-to-real gap: dynamics, contact, estimation, solvers.

Structured a dual approach: improving simulator fidelity and robust policy training.

Provided a strategic framework for developing resilient sim-to-real transfer methods.

Abstract

This chapter addresses the critical challenge of simulation-to-reality (sim-to-real) transfer for deep reinforcement learning (DRL) in bipedal locomotion. After contextualizing the problem within various control architectures, we dissect the ``curse of simulation'' by analyzing the primary sources of sim-to-real gap: robot dynamics, contact modeling, state estimation, and numerical solvers. Building on this diagnosis, we structure the solutions around two complementary philosophies. The first is to shrink the gap through model-centric strategies that systematically improve the simulator's physical fidelity. The second is to harden the policy, a complementary approach that uses in-simulation robustness training and post-deployment adaptation to make the policy inherently resilient to model inaccuracies. The chapter concludes by synthesizing these philosophies into a strategic framework, providing a clear roadmap for developing and evaluating robust sim-to-real solutions.