Using Simulation Optimization to Improve Zero-shot Policy Transfer of Quadrotors

TL;DR

This paper presents a data-driven simulation optimization method to enhance zero-shot transfer of neural network-based quadrotor control policies from simulation to real-world, emphasizing the importance of accurate simulation for low-level RL controllers.

Contribution

It introduces a simulation parameter optimization approach that improves the transferability of control policies, especially for low-level controllers trained with reinforcement learning.

Findings

Low-level RL controllers require more accurate simulation parameters.

Higher-level control policies are more robust to simulation inaccuracies.

Real-world experiments validate the effectiveness of the optimized simulation.

Abstract

In this work, we propose a data-driven approach to optimize the parameters of a simulation such that control policies can be directly transferred from simulation to a real-world quadrotor. Our neural network-based policies take only onboard sensor data as input and run entirely on the embedded hardware. In extensive real-world experiments, we compare low-level Pulse-Width Modulated control with higher-level control structures such as Attitude Rate and Attitude, which utilize Proportional-Integral-Derivative controllers to output motor commands. Our experiments show that low-level controllers trained with reinforcement learning require a more accurate simulation than higher-level control policies.