Risk-Sensitive Bayesian Games for Multi-Agent Reinforcement Learning under Policy Uncertainty

TL;DR

This paper introduces risk-sensitive algorithms for Bayesian games in multi-agent reinforcement learning, focusing on uncertainty over agent types and demonstrating improved performance over risk-neutral methods.

Contribution

It develops risk-sensitive variants of existing algorithms like IBR, FP, and DAPG to handle type uncertainty in stochastic games, advancing the field of risk-aware multi-agent RL.

Findings

Risk-sensitive DAPG outperforms risk-neutral algorithms in experiments.

Focus on type uncertainty offers new insights into stochastic game risks.

Algorithms improve social welfare in general-sum stochastic games.

Abstract

In stochastic games with incomplete information, the uncertainty is evoked by the lack of knowledge about a player's own and the other players' types, i.e. the utility function and the policy space, and also the inherent stochasticity of different players' interactions. In existing literature, the risk in stochastic games has been studied in terms of the inherent uncertainty evoked by the variability of transitions and actions. In this work, we instead focus on the risk associated with the \textit{uncertainty over types}. We contrast this with the multi-agent reinforcement learning framework where the other agents have fixed stationary policies and investigate risk-sensitiveness due to the uncertainty about the other agents' adaptive policies. We propose risk-sensitive versions of existing algorithms proposed for risk-neutral stochastic games, such as Iterated Best Response (IBR), Fictitious Play (FP) and a general multi-objective gradient approach using dual ascent (DAPG). Our experimental analysis shows that risk-sensitive DAPG performs better than competing algorithms for both social welfare and general-sum stochastic games.