Hovering Flight of Soft-Actuated Insect-Scale Micro Aerial Vehicles using Deep Reinforcement Learning

TL;DR

This paper develops a deep reinforcement learning controller for soft-actuated insect-scale micro aerial vehicles, enabling robust, zero-shot hovering flights despite system delays and uncertainties, with successful real-world deployment.

Contribution

It introduces a novel combined approach of behavior cloning with state-action re-matching and RL fine-tuning for controlling soft IMAVs, achieving first end-to-end deep RL flight.

Findings

Deep RL controller enables stable hovering flights.

First successful real-world deep RL flight on soft IMAVs.

Achieves 50-second hover with low positional error.

Abstract

Soft-actuated insect-scale micro aerial vehicles (IMAVs) pose unique challenges for designing robust and computationally efficient controllers. At the millimeter scale, fast robot dynamics ($\sim$ms), together with system delay, model uncertainty, and external disturbances significantly affect flight performances. Here, we design a deep reinforcement learning (RL) controller that addresses system delay and uncertainties. To initialize this neural network (NN) controller, we propose a modified behavior cloning (BC) approach with state-action re-matching to account for delay and domain-randomized expert demonstration to tackle uncertainty. Then we apply proximal policy optimization (PPO) to fine-tune the policy during RL, enhancing performance and smoothing commands. In simulations, our modified BC substantially increases the mean reward compared to baseline BC; and RL with PPO improves flight quality and reduces command fluctuations. We deploy this controller on two different insect-scale aerial robots that weigh 720 mg and 850 mg, respectively. The robots demonstrate multiple successful zero-shot hovering flights, with the longest lasting 50 seconds and root-mean-square errors of 1.34 cm in lateral direction and 0.05 cm in altitude, marking the first end-to-end deep RL-based flight on soft-driven IMAVs.