Decentralized Aerial Manipulation of a Cable-Suspended Load using Multi-Agent Reinforcement Learning

TL;DR

This paper introduces a scalable, decentralized multi-agent reinforcement learning approach for MAV teams to manipulate cable-suspended loads in real-world scenarios without requiring global state information or inter-agent communication.

Contribution

It presents the first decentralized MARL-based control method enabling MAV teams to perform 6-DoF load manipulation with high scalability and robustness, using implicit communication through load observations.

Findings

Achieved reliable load manipulation with no global state or inter-MAV communication.

Demonstrated successful sim-to-real transfer despite cable tension uncertainties.

Showed robustness to in-flight loss of a MAV and heterogeneous control policies.

Abstract

This paper presents the first decentralized method to enable real-world 6-DoF manipulation of a cable-suspended load using a team of Micro-Aerial Vehicles (MAVs). Our method leverages multi-agent reinforcement learning (MARL) to train an outer-loop control policy for each MAV. Unlike state-of-the-art controllers that utilize a centralized scheme, our policy does not require global states, inter-MAV communications, nor neighboring MAV information. Instead, agents communicate implicitly through load pose observations alone, which enables high scalability and flexibility. It also significantly reduces computing costs during inference time, enabling onboard deployment of the policy. In addition, we introduce a new action space design for the MAVs using linear acceleration and body rates. This choice, combined with a robust low-level controller, enables reliable sim-to-real transfer despite significant uncertainties caused by cable tension during dynamic 3D motion. We validate our method in various real-world experiments, including full-pose control under load model uncertainties, showing setpoint tracking performance comparable to the state-of-the-art centralized method. We also demonstrate cooperation amongst agents with heterogeneous control policies, and robustness to the complete in-flight loss of one MAV. Videos of experiments: https://autonomousrobots.nl/paper_websites/aerial-manipulation-marl