Understanding Individual Decision-Making in Multi-Agent Reinforcement Learning: A Dynamical Systems Approach

TL;DR

This paper introduces a novel dynamical systems framework to analyze individual decision-making in multi-agent reinforcement learning, accounting for stochasticity and interactions, thus offering deeper insights into system stability and behavior.

Contribution

It proposes modeling MARL as coupled stochastic dynamical systems, enabling rigorous analysis of individual agent stability and sensitivity, which was not previously feasible.

Findings

Framework captures agent interactions and environmental noise.

Allows analysis of stability and sensitivity of individual behaviors.

Provides insights for safe and reliable multi-agent system design.

Abstract

Analysing learning behaviour in Multi-Agent Reinforcement Learning (MARL) environments is challenging, in particular with respect to \textit{individual} decision-making. Practitioners frequently tend to study or compare MARL algorithms from a qualitative perspective largely due to the inherent stochasticity in practical algorithms arising from random dithering exploration strategies, environment transition noise, and stochastic gradient updates to name a few. Traditional analytical approaches, such as replicator dynamics, often rely on mean-field approximations to remove stochastic effects, but this simplification, whilst able to provide general overall trends, might lead to dissonance between analytical predictions and actual realisations of individual trajectories. In this paper, we propose a novel perspective on MARL systems by modelling them as \textit{coupled stochastic dynamical systems}, capturing both agent interactions and environmental characteristics. Leveraging tools from dynamical systems theory, we analyse the stability and sensitivity of agent behaviour at individual level, which are key dimensions for their practical deployments, for example, in presence of strict safety requirements. This framework allows us, for the first time, to rigorously study MARL dynamics taking into consideration their inherent stochasticity, providing a deeper understanding of system behaviour and practical insights for the design and control of multi-agent learning processes.