Evolutionary Multi-agent Reinforcement Learning in Group Social Dilemmas

TL;DR

This paper explores how evolutionary multi-agent reinforcement learning influences cooperation in social dilemmas like public goods games, combining game theory and learning dynamics to understand AI behavior evolution.

Contribution

It introduces an integrated approach analyzing the impact of learning parameters and evolutionary pressures on cooperation in multi-agent Q-learning within social dilemmas.

Findings

Selection for different exploration levels affects cooperation outcomes.

A condition separates attraction basins in social dilemma games.

Theoretical insights into evolutionary Q-learning enhance understanding of AI social behavior.

Abstract

Reinforcement learning (RL) is a powerful machine learning technique that has been successfully applied to a wide variety of problems. However, it can be unpredictable and produce suboptimal results in complicated learning environments. This is especially true when multiple agents learn simultaneously, which creates a complex system that is often analytically intractable. Our work considers the fundamental framework of Q-learning in Public Goods Games, where RL individuals must work together to achieve a common goal. This setting allows us to study the tragedy of the commons and free rider effects in AI cooperation, an emerging field with potential to resolve challenging obstacles to the wider application of artificial intelligence. While this social dilemma has been mainly investigated through traditional and evolutionary game theory, our approach bridges the gap between these two by studying agents with an intermediate level of intelligence. Specifically, we consider the influence of learning parameters on cooperation levels in simulations and a limiting system of differential equations, as well as the effect of evolutionary pressures on exploration rate in both of these models. We find selection for higher and lower levels of exploration, as well as attracting values, and a condition that separates these in a restricted class of games. Our work enhances the theoretical understanding of evolutionary Q-learning, and extends our knowledge of the evolution of machine behavior in social dilemmas.