Eco-evolutionary Dynamics of Non-episodic Neuroevolution in Large Multi-agent Environments

TL;DR

This paper introduces a continuous neuroevolution method in a large, multi-agent environment with complex resource dynamics, demonstrating sustainable foraging strategies without environment resets.

Contribution

It presents a novel non-episodic neuroevolution approach in multi-agent settings with ecological validity, implemented efficiently in JAX for fast GPU simulation.

Findings

Neuroevolution can operate continuously in complex multi-agent environments.

Agents develop sustainable foraging strategies through ecological and evolutionary interactions.

The system runs efficiently on GPU, enabling large-scale simulations.

Abstract

Neuroevolution (NE) has recently proven a competitive alternative to learning by gradient descent in reinforcement learning tasks. However, the majority of NE methods and associated simulation environments differ crucially from biological evolution: the environment is reset to initial conditions at the end of each generation, whereas natural environments are continuously modified by their inhabitants; agents reproduce based on their ability to maximize rewards within a population, while biological organisms reproduce and die based on internal physiological variables that depend on their resource consumption; simulation environments are primarily single-agent while the biological world is inherently multi-agent and evolves alongside the population. In this work we present a method for continuously evolving adaptive agents without any environment or population reset. The environment is a large grid world with complex spatiotemporal resource generation, containing many agents that are each controlled by an evolvable recurrent neural network and locally reproduce based on their internal physiology. The entire system is implemented in JAX, allowing very fast simulation on a GPU. We show that NE can operate in an ecologically-valid non-episodic multi-agent setting, finding sustainable collective foraging strategies in the presence of a complex interplay between ecological and evolutionary dynamics.