ForMIC: Foraging via Multiagent RL with Implicit Communication

TL;DR

This paper introduces ForMIC, a multi-agent reinforcement learning approach that enables agents to communicate implicitly through their shared environment, improving foraging efficiency in complex, dynamic settings.

Contribution

The work develops novel training techniques for stigmergic multi-agent policies, including curriculum learning, action filtering, and the use of non-learning agents to enhance stability and performance.

Findings

ForMIC outperforms existing algorithms in various foraging scenarios.

The approach is robust to changes in team size and resource placement.

Stable training of stigmergic policies is achieved through the proposed methods.

Abstract

Multi-agent foraging (MAF) involves distributing a team of agents to search an environment and extract resources from it. Nature provides several examples of highly effective foragers, where individuals within the foraging collective use biological markers (e.g., pheromones) to communicate critical information to others via the environment. In this work, we propose ForMIC, a distributed reinforcement learning MAF approach that endows agents with implicit communication abilities via their shared environment. However, learning efficient policies with stigmergic interactions is highly nontrivial, since agents need to perform well to send each other useful signals, but also need to sense others' signals to perform well. In this work, we develop several key learning techniques for training policies with stigmergic interactions, where such a circular dependency is present. By relying on clever curriculum learning design, action filtering, and the introduction of non-learning agents to increase the agent density at training time at low computational cost, we develop a minimal learning framework that leads to the stable training of efficient stigmergic policies. We present simulation results which demonstrate that our learned policy outperforms existing state-of-the-art MAF algorithms in a set of experiments that vary team size, number and placement of resources, and key environmental dynamics not seen at training time.