Learning to Fly -- a Gym Environment with PyBullet Physics for Reinforcement Learning of Multi-agent Quadcopter Control

TL;DR

This paper introduces an open-source multi-agent quadcopter simulation environment using PyBullet, enabling realistic physics, vision-based RL, and control experiments to advance research in drone control and learning algorithms.

Contribution

It presents a novel, realistic, multi-agent quadcopter environment compatible with OpenAI Gym, supporting RL and control methods with physics-based interactions.

Findings

Demonstrated control and RL tasks with the environment

Showcased multi-robot flight with aerodynamic effects

Provided a platform for benchmarking control and learning algorithms

Abstract

Robotic simulators are crucial for academic research and education as well as the development of safety-critical applications. Reinforcement learning environments -- simple simulations coupled with a problem specification in the form of a reward function -- are also important to standardize the development (and benchmarking) of learning algorithms. Yet, full-scale simulators typically lack portability and parallelizability. Vice versa, many reinforcement learning environments trade-off realism for high sample throughputs in toy-like problems. While public data sets have greatly benefited deep learning and computer vision, we still lack the software tools to simultaneously develop -- and fairly compare -- control theory and reinforcement learning approaches. In this paper, we propose an open-source OpenAI Gym-like environment for multiple quadcopters based on the Bullet physics engine. Its multi-agent and vision based reinforcement learning interfaces, as well as the support of realistic collisions and aerodynamic effects, make it, to the best of our knowledge, a first of its kind. We demonstrate its use through several examples, either for control (trajectory tracking with PID control, multi-robot flight with downwash, etc.) or reinforcement learning (single and multi-agent stabilization tasks), hoping to inspire future research that combines control theory and machine learning.