Risk Sensitivity in Markov Games and Multi-Agent Reinforcement Learning: A Systematic Review

TL;DR

This paper systematically reviews how risk sensitivity is incorporated into Markov games and multi-agent reinforcement learning, highlighting recent theoretical and practical developments and future research directions.

Contribution

It provides a comprehensive overview of risk measures in MG and MARL, detailing their mathematical foundations and surveying recent advances and trends.

Findings

Various risk measures are used in MG and MARL models.

Recent research trends focus on theoretical and applied risk-sensitive methods.

Future directions include developing new risk models and applications.

Abstract

Markov games (MGs) and multi-agent reinforcement learning (MARL) are studied to model decision making in multi-agent systems. Traditionally, the objective in MG and MARL has been risk-neutral, i.e., agents are assumed to optimize a performance metric such as expected return, without taking into account subjective or cognitive preferences of themselves or of other agents. However, ignoring such preferences leads to inaccurate models of decision making in many real-world scenarios in finance, operations research, and behavioral economics. Therefore, when these preferences are present, it is necessary to incorporate a suitable measure of risk into the optimization objective of agents, which opens the door to risk-sensitive MG and MARL. In this paper, we systemically review the literature on risk sensitivity in MG and MARL that has been growing in recent years alongside other areas of reinforcement learning and game theory. We define and mathematically describe different risk measures used in MG and MARL and individually for each measure, discuss articles that incorporate it. Finally, we identify recent trends in theoretical and applied works in the field and discuss possible directions of future research.