Environmental effects on emergent strategy in micro-scale multi-agent reinforcement learning

TL;DR

This paper investigates how temperature influences the emergence of strategies in multi-agent reinforcement learning within micro-scale environments, revealing new strategies at higher temperatures and providing a tool for further study.

Contribution

It introduces a study of temperature effects on MARL in microscopic environments and presents a new Python package for analyzing microscopic agents with RL.

Findings

Higher temperatures lead to new emergent strategies in MARL agents.

Temperature significantly affects the efficacy of strategies in micro-scale tasks.

The study provides insights for better training strategies bridging simulation and real micro-environments.

Abstract

Multi-Agent Reinforcement Learning (MARL) is a promising candidate for realizing efficient control of microscopic particles, of which micro-robots are a subset. However, the microscopic particles' environment presents unique challenges, such as Brownian motion at sufficiently small length-scales. In this work, we explore the role of temperature in the emergence and efficacy of strategies in MARL systems using particle-based Langevin molecular dynamics simulations as a realistic representation of micro-scale environments. To this end, we perform experiments on two different multi-agent tasks in microscopic environments at different temperatures, detecting the source of a concentration gradient and rotation of a rod. We find that at higher temperatures, the RL agents identify new strategies for achieving these tasks, highlighting the importance of understanding this regime and providing insight into optimal training strategies for bridging the generalization gap between simulation and reality. We also introduce a novel Python package for studying microscopic agents using reinforcement learning (RL) to accompany our results.