Retro-RL: Reinforcing Nominal Controller With Deep Reinforcement Learning for Tilting-Rotor Drones

TL;DR

This paper introduces a hybrid control architecture for tilting-rotor drones that combines a nominal controller with a deep reinforcement learning policy, enhancing robustness and stability in complex tasks like wall climbing.

Contribution

It proposes an uncertainty-aware control mixer that integrates a learned policy with a nominal controller, ensuring stability and robustness for real-world drone applications.

Findings

The hybrid approach outperforms conventional controllers in real-world tests.

The method maintains stability while leveraging deep RL for robustness.

Extensive domain randomization improves policy generalization.

Abstract

Studies that broaden drone applications into complex tasks require a stable control framework. Recently, deep reinforcement learning (RL) algorithms have been exploited in many studies for robot control to accomplish complex tasks. Unfortunately, deep RL algorithms might not be suitable for being deployed directly into a real-world robot platform due to the difficulty in interpreting the learned policy and lack of stability guarantee, especially for a complex task such as a wall-climbing drone. This paper proposes a novel hybrid architecture that reinforces a nominal controller with a robust policy learned using a model-free deep RL algorithm. The proposed architecture employs an uncertainty-aware control mixer to preserve guaranteed stability of a nominal controller while using the extended robust performance of the learned policy. The policy is trained in a simulated environment with thousands of domain randomizations to achieve robust performance over diverse uncertainties. The performance of the proposed method was verified through real-world experiments and then compared with a conventional controller and the state-of-the-art learning-based controller trained with a vanilla deep RL algorithm.