Learning Density-Based Correlated Equilibria for Markov Games

TL;DR

This paper introduces Density-Based Correlated Equilibria (DBCE), a novel approach that explicitly incorporates state density requirements into the equilibrium concept for Markov games, improving coordination with safety and fairness considerations.

Contribution

It proposes a new equilibrium concept, DBCE, and a corresponding policy iteration algorithm to explicitly meet state density constraints in multi-agent systems.

Findings

DBCE effectively incorporates state density requirements.

The proposed algorithm outperforms existing methods in relevant scenarios.

Experiments demonstrate improved safety and fairness in multi-agent coordination.

Abstract

Correlated Equilibrium (CE) is a well-established solution concept that captures coordination among agents and enjoys good algorithmic properties. In real-world multi-agent systems, in addition to being in an equilibrium, agents' policies are often expected to meet requirements with respect to safety, and fairness. Such additional requirements can often be expressed in terms of the state density which measures the state-visitation frequencies during the course of a game. However, existing CE notions or CE-finding approaches cannot explicitly specify a CE with particular properties concerning state density; they do so implicitly by either modifying reward functions or using value functions as the selection criteria. The resulting CE may thus not fully fulfil the state-density requirements. In this paper, we propose Density-Based Correlated Equilibria (DBCE), a new notion of CE that explicitly takes state density as selection criterion. Concretely, we instantiate DBCE by specifying different state-density requirements motivated by real-world applications. To compute DBCE, we put forward the Density Based Correlated Policy Iteration algorithm for the underlying control problem. We perform experiments on various games where results demonstrate the advantage of our CE-finding approach over existing methods in scenarios with state-density concerns.