Approximate Nash Equilibrium Learning for n-Player Markov Games in Dynamic Pricing

TL;DR

This paper introduces a model-free, neural network-based method for learning approximate Nash equilibria in multi-agent Markov games, especially applied to dynamic pricing where exact solutions are computationally infeasible.

Contribution

A novel gradient-free optimization approach combined with neural networks to estimate approximate Nash equilibria in complex, multi-agent Markov games.

Findings

Successfully learned approximate Nash equilibria in dynamic pricing scenarios.

Demonstrated the method's effectiveness in high-dimensional, multi-agent environments.

Provided a scalable approach to equilibrium computation where exact solutions are intractable.

Abstract

We investigate Nash equilibrium learning in a competitive Markov Game (MG) environment, where multiple agents compete, and multiple Nash equilibria can exist. In particular, for an oligopolistic dynamic pricing environment, exact Nash equilibria are difficult to obtain due to the curse-of-dimensionality. We develop a new model-free method to find approximate Nash equilibria. Gradient-free black box optimization is then applied to estimate $\epsilon$, the maximum reward advantage of an agent unilaterally deviating from any joint policy, and to also estimate the $\epsilon$-minimizing policy for any given state. The policy-$\epsilon$ correspondence and the state to $\epsilon$-minimizing policy are represented by neural networks, the latter being the Nash Policy Net. During batch update, we perform Nash Q learning on the system, by adjusting the action probabilities using the Nash Policy Net. We demonstrate that an approximate Nash equilibrium can be learned, particularly in the dynamic pricing domain where exact solutions are often intractable.