Multi-Agent Generative Adversarial Interactive Self-Imitation Learning for AUV Formation Control and Obstacle Avoidance

TL;DR

This paper introduces MAGAISIL, a novel learning method for multi-AUVs that improves formation control and obstacle avoidance by iteratively replacing sub-optimal demonstrations with self-generated trajectories, surpassing prior imitation learning methods.

Contribution

The paper proposes MAGAISIL, an enhanced multi-agent imitation learning algorithm that enables AUVs to improve policies beyond initial demonstrations through self-imitation and human guidance.

Findings

MAGAISIL outperforms MAGAIL with sub-optimal demonstrations.

AUV policies trained with MAGAISIL match or exceed those trained with optimal demonstrations.

The method adapts well to complex, varied tasks.

Abstract

Multiple autonomous underwater vehicles (multi-AUV) can cooperatively accomplish tasks that a single AUV cannot complete. Recently, multi-agent reinforcement learning has been introduced to control of multi-AUV. However, designing efficient reward functions for various tasks of multi-AUV control is difficult or even impractical. Multi-agent generative adversarial imitation learning (MAGAIL) allows multi-AUV to learn from expert demonstration instead of pre-defined reward functions, but suffers from the deficiency of requiring optimal demonstrations and not surpassing provided expert demonstrations. This paper builds upon the MAGAIL algorithm by proposing multi-agent generative adversarial interactive self-imitation learning (MAGAISIL), which can facilitate AUVs to learn policies by gradually replacing the provided sub-optimal demonstrations with self-generated good trajectories selected by a human trainer. Our experimental results in a multi-AUV formation control and obstacle avoidance task on the Gazebo platform with AUV simulator of our lab show that AUVs trained via MAGAISIL can surpass the provided sub-optimal expert demonstrations and reach a performance close to or even better than MAGAIL with optimal demonstrations. Further results indicate that AUVs' policies trained via MAGAISIL can adapt to complex and different tasks as well as MAGAIL learning from optimal demonstrations.