Reinforcement Learning for UAV control with Policy and Reward Shaping

TL;DR

This paper explores the use of reward and policy shaping techniques in reinforcement learning to improve UAV control, demonstrating faster training times and reduced variability despite lower rewards.

Contribution

It introduces a combined reward and policy shaping approach for UAV control in reinforcement learning and evaluates its effects in simulated scenarios with and without obstacles.

Findings

Combined shaping techniques reduce training time.

Lower reward but more consistent training outcomes.

Faster convergence in obstacle scenarios.

Abstract

In recent years, unmanned aerial vehicle (UAV) related technology has expanded knowledge in the area, bringing to light new problems and challenges that require solutions. Furthermore, because the technology allows processes usually carried out by people to be automated, it is in great demand in industrial sectors. The automation of these vehicles has been addressed in the literature, applying different machine learning strategies. Reinforcement learning (RL) is an automation framework that is frequently used to train autonomous agents. RL is a machine learning paradigm wherein an agent interacts with an environment to solve a given task. However, learning autonomously can be time consuming, computationally expensive, and may not be practical in highly-complex scenarios. Interactive reinforcement learning allows an external trainer to provide advice to an agent while it is learning a task. In this study, we set out to teach an RL agent to control a drone using reward-shaping and policy-shaping techniques simultaneously. Two simulated scenarios were proposed for the training; one without obstacles and one with obstacles. We also studied the influence of each technique. The results show that an agent trained simultaneously with both techniques obtains a lower reward than an agent trained using only a policy-based approach. Nevertheless, the agent achieves lower execution times and less dispersion during training.