Efficient Environment Design for Multi-Robot Navigation via Continuous Control

TL;DR

This paper presents a customizable environment for multi-robot navigation in continuous spaces, modeled as an MDP, and evaluates various RL algorithms in simulation and real-world-like settings to improve efficiency and robustness.

Contribution

It introduces a formalized, adaptable environment for multi-robot navigation as an MDP and assesses multiple RL methods, bridging simulation and real-world applications.

Findings

RL algorithms successfully learned navigation policies in the environment.

The environment's design facilitates testing robustness in uncertain real-world scenarios.

Different RL methods showed varying levels of efficiency and performance.

Abstract

Multi-robot navigation and path planning in continuous state and action spaces with uncertain environments remains an open challenge. Deep Reinforcement Learning (RL) is one of the most popular paradigms for solving this task, but its real-world application has been limited due to sample inefficiency and long training periods. Moreover, the existing works using RL for multi-robot navigation lack formal guarantees while designing the environment. In this paper, we introduce an efficient and highly customizable environment for continuous-control multi-robot navigation, where the robots must visit a set of regions of interest (ROIs) by following the shortest paths. The task is formally modeled as a Markov Decision Process (MDP). We describe the multi-robot navigation task as an optimization problem and relate it to finding an optimal policy for the MDP. We crafted several variations of the environment and measured the performance using both gradient and non-gradient based RL methods: A2C, PPO, TRPO, TQC, CrossQ and ARS. To show real-world applicability, we deployed our environment to a 3-D agricultural field with uncertainties using the CoppeliaSim robot simulator and measured the robustness by running inference on the learned models. We believe our work will guide the researchers on how to develop MDP-based environments that are applicable to real-world systems and solve them using the existing state-of-the-art RL methods with limited resources and within reasonable time periods.