Multi-Agent Reinforcement Learning for Visibility-based Persistent Monitoring

TL;DR

This paper introduces a multi-agent reinforcement learning approach using graph attention and PPO to optimize persistent monitoring by robots with limited views, demonstrating improved policies over baselines.

Contribution

It proposes a novel MA-G-PPO algorithm that effectively coordinates multiple robots for visibility-based monitoring using local and global information.

Findings

MA-G-PPO outperforms non-RL baseline in most scenarios

Sharing information among agents enhances policy effectiveness

Emergent behaviors are observed in the learned policies

Abstract

The Visibility-based Persistent Monitoring (VPM) problem seeks to find a set of trajectories (or controllers) for robots to persistently monitor a changing environment. Each robot has a sensor, such as a camera, with a limited field-of-view that is obstructed by obstacles in the environment. The robots may need to coordinate with each other to ensure no point in the environment is left unmonitored for long periods of time. We model the problem such that there is a penalty that accrues every time step if a point is left unmonitored. However, the dynamics of the penalty are unknown to us. We present a Multi-Agent Reinforcement Learning (MARL) algorithm for the VPM problem. Specifically, we present a Multi-Agent Graph Attention Proximal Policy Optimization (MA-G-PPO) algorithm that takes as input the local observations of all agents combined with a low resolution global map to learn a policy for each agent. The graph attention allows agents to share their information with others leading to an effective joint policy. Our main focus is to understand how effective MARL is for the VPM problem. We investigate five research questions with this broader goal. We find that MA-G-PPO is able to learn a better policy than the non-RL baseline in most cases, the effectiveness depends on agents sharing information with each other, and the policy learnt shows emergent behavior for the agents.