Multi-Agent Reinforcement Learning for Dynamic Ocean Monitoring by a Swarm of Buoys

TL;DR

This paper introduces two multi-agent reinforcement learning methods for dynamic ocean monitoring using a swarm of autonomous buoys, improving area coverage efficiency in non-stationary environments.

Contribution

It presents a novel structured MARL approach for marine area coverage, including a modified MADDPG method that enhances collective behavior and coverage performance.

Findings

Coverage-range-based MARL outperforms swarm-based MARL in convergence and coverage.

Both MARL methods outperform naive swarming strategies.

Structured learning improves adaptability in non-stationary environments.

Abstract

Autonomous marine environmental monitoring problem traditionally encompasses an area coverage problem which can only be effectively carried out by a multi-robot system. In this paper, we focus on robotic swarms that are typically operated and controlled by means of simple swarming behaviors obtained from a subtle, yet ad hoc combination of bio-inspired strategies. We propose a novel and structured approach for area coverage using multi-agent reinforcement learning (MARL) which effectively deals with the non-stationarity of environmental features. Specifically, we propose two dynamic area coverage approaches: (1) swarm-based MARL, and (2) coverage-range-based MARL. The former is trained using the multi-agent deep deterministic policy gradient (MADDPG) approach whereas, a modified version of MADDPG is introduced for the latter with a reward function that intrinsically leads to a collective behavior. Both methods are tested and validated with different geometric shaped regions with equal surface area (square vs. rectangle) yielding acceptable area coverage, and benefiting from the structured learning in non-stationary environments. Both approaches are advantageous compared to a na\"{i}ve swarming method. However, coverage-range-based MARL outperforms the swarm-based MARL with stronger convergence features in learning criteria and higher spreading of agents for area coverage.