Guided Deep Reinforcement Learning for Swarm Systems

TL;DR

This paper presents a guided deep reinforcement learning approach for controlling swarm agents with limited sensing, using a central critic with global state access to improve policy learning for tasks like formation and target search.

Contribution

It introduces a novel actor-critic method where the critic has global state access during training, enhancing learning for decentralized swarm control policies.

Findings

Effective in simulated tasks of swarm formation and target localization

Demonstrates improved policy learning with guided critic approach

Applicable to cooperative agents with limited sensing capabilities

Abstract

In this paper, we investigate how to learn to control a group of cooperative agents with limited sensing capabilities such as robot swarms. The agents have only very basic sensor capabilities, yet in a group they can accomplish sophisticated tasks, such as distributed assembly or search and rescue tasks. Learning a policy for a group of agents is difficult due to distributed partial observability of the state. Here, we follow a guided approach where a critic has central access to the global state during learning, which simplifies the policy evaluation problem from a reinforcement learning point of view. For example, we can get the positions of all robots of the swarm using a camera image of a scene. This camera image is only available to the critic and not to the control policies of the robots. We follow an actor-critic approach, where the actors base their decisions only on locally sensed information. In contrast, the critic is learned based on the true global state. Our algorithm uses deep reinforcement learning to approximate both the Q-function and the policy. The performance of the algorithm is evaluated on two tasks with simple simulated 2D agents: 1) finding and maintaining a certain distance to each others and 2) locating a target.