Evaluating Collaborative Autonomy in Opposed Environments using Maritime Capture-the-Flag Competitions

TL;DR

This study evaluates multi-agent AI methods for autonomous USV teams in adversarial Capture-the-Flag competitions, comparing behavior-based and deep reinforcement learning approaches in real-world maritime scenarios.

Contribution

It introduces a novel testing framework combining real-world USV competitions with simulated RL training environments, and compares the effectiveness of different cooperative algorithms.

Findings

Rule-based cooperation outperformed deep RL in current tests.

Deep RL methods show potential for future improvements.

Integration of simulation and real-world testing enhances autonomous maritime security.

Abstract

The objective of this work is to evaluate multi-agent artificial intelligence methods when deployed on teams of unmanned surface vehicles (USV) in an adversarial environment. Autonomous agents were evaluated in real-world scenarios using the Aquaticus test-bed, which is a Capture-the-Flag (CTF) style competition involving teams of USV systems. Cooperative teaming algorithms of various foundations in behavior-based optimization and deep reinforcement learning (RL) were deployed on these USV systems in two versus two teams and tested against each other during a competition period in the fall of 2023. Deep reinforcement learning applied to USV agents was achieved via the Pyquaticus test bed, a lightweight gymnasium environment that allows simulated CTF training in a low-level environment. The results of the experiment demonstrate that rule-based cooperation for behavior-based agents outperformed those trained in Deep-reinforcement learning paradigms as implemented in these competitions. Further integration of the Pyquaticus gymnasium environment for RL with MOOS-IvP in terms of configuration and control schema will allow for more competitive CTF games in future studies. As the development of experimental deep RL methods continues, the authors expect that the competitive gap between behavior-based autonomy and deep RL will be reduced. As such, this report outlines the overall competition, methods, and results with an emphasis on future works such as reward shaping and sim-to-real methodologies and extending rule-based cooperation among agents to react to safety and security events in accordance with human experts intent/rules for executing safety and security processes.