Markov Potential Game and Multi-Agent Reinforcement Learning for Autonomous Driving

TL;DR

This paper introduces conditions for constructing Markov potential games suitable for autonomous driving, enabling decentralized multi-agent reinforcement learning with neural networks, and evaluates policies in traffic scenarios.

Contribution

It provides a framework for creating Markov potential games tailored for autonomous driving, facilitating reliable multi-agent RL with neural networks.

Findings

Conditions for MG construction applicable to driving scenarios

Neural network policy enables decentralized decision-making

Trained policies perform well in simulated and real traffic data

Abstract

Autonomous driving (AD) requires safe and reliable decision-making among interacting agents, e.g., vehicles, bicycles, and pedestrians. Multi-agent reinforcement learning (MARL) modeled by Markov games (MGs) provides a suitable framework to characterize such agents' interactions during decision-making. Nash equilibria (NEs) are often the desired solution in an MG. However, it is typically challenging to compute an NE in general-sum games, unless the game is a Markov potential game (MPG), which ensures the NE attainability under a few learning algorithms such as gradient play. However, it has been an open question how to construct an MPG and whether these construction rules are suitable for AD applications. In this paper, we provide sufficient conditions under which an MG is an MPG and show that these conditions can accommodate general driving objectives for autonomous vehicles (AVs) using highway forced merge scenarios as illustrative examples. A parameter-sharing neural network (NN) structure is designed to enable decentralized policy execution. The trained driving policy from MPGs is evaluated in both simulated and naturalistic traffic datasets. Comparative studies with single-agent RL and with human drivers whose behaviors are recorded in the traffic datasets are reported, respectively.