Deep Q-Learning for Nash Equilibria: Nash-DQN

TL;DR

This paper introduces Nash-DQN, a deep Q-learning algorithm designed to efficiently learn Nash equilibria in general-sum stochastic games, overcoming limitations of prior methods restricted to simpler game types.

Contribution

The paper presents a novel deep Q-learning approach that uses a local linear-quadratic expansion to learn Nash equilibria in complex multi-agent stochastic games.

Findings

Successfully applied to learning optimal trading strategies in electronic markets.

Demonstrates data efficiency and flexibility in complex game environments.

Extends reinforcement learning to general-sum stochastic games beyond zero-sum cases.

Abstract

Model-free learning for multi-agent stochastic games is an active area of research. Existing reinforcement learning algorithms, however, are often restricted to zero-sum games, and are applicable only in small state-action spaces or other simplified settings. Here, we develop a new data efficient Deep-Q-learning methodology for model-free learning of Nash equilibria for general-sum stochastic games. The algorithm uses a local linear-quadratic expansion of the stochastic game, which leads to analytically solvable optimal actions. The expansion is parametrized by deep neural networks to give it sufficient flexibility to learn the environment without the need to experience all state-action pairs. We study symmetry properties of the algorithm stemming from label-invariant stochastic games and as a proof of concept, apply our algorithm to learning optimal trading strategies in competitive electronic markets.

Figures

Tables

Table 1. Table 1 : Asset price process, price impact, and risk preference parameters.

$\kappa$	$\theta$	$\sigma$	$\gamma$	$\rho$	$\eta$	$b_1$	$b_2$	$b_3$	$\Delta T$
0.1	10	0.01	0.02	0.5	0.05	0.1	0.1	0	0.5

Equations

$${\mathcal{R}}_i(x;{\pi }_i,{\mathbf{\pi }}_{-i})=\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }_i^{-t}{r}_i(x_t,{\pi }_{i,t},{\mathbf{\pi }}_{-i,t})\right],$$

$${\mathcal{R}}_i\left(x;{\pi }_i,{\mathbf{\pi }}_{-i}^{\ast }\right)\le {\mathcal{R}}_i\left(x;{\pi }_i^{\ast },{\mathbf{\pi }}_{-i}^{\ast }\right)$$

$${\mathcal{R}}_i(x;{\pi }_i^{\ast },{\mathbf{\pi }}_{-i}^{\ast })=\underset{u\in {\mathcal{U}}_i}{\max }\left\{r_i(x,u,{\mathbf{\pi }}_{-i}^{\ast }(x))+{\gamma }_i\underset{x^{\prime }\sim p(\cdot \mid x,\mathbf{u})}{\mathbb{E}}\left[{\mathcal{R}}_i(x^{\prime };{\pi }_{i,1}^{\ast },{\mathbf{\pi }}_{-i,1}^{\ast })\right]\right\}.$$

$$\mathbf{Q}(x;\mathbf{u})=\mathbf{r}(x;\mathbf{u})+{\gamma }_i\underset{x^{\prime }\sim p(\cdot \mid x,\mathbf{u})}{\mathbb{E}}\left[\mathbf{V}(x^{\prime })\right],$$

$$f_i\left({\omega }_i,{\mathbf{u}}_{-i}^{\ast }\right)\le f_i\left(u_i,{\mathbf{u}}_{-i}^{\ast }\right),\forall {\omega }_i\in {\mathcal{U}}_i,\text{ and }\forall i\in \mathfrak{N}.$$

$$\mathbf{V}(x)$$

$$\underset{\begin{array}{c}x\sim \rho \\ x^{\prime }\sim p(\cdot \mid x,\mathbf{u})\end{array}}{\mathbb{E}}\left[{\Vert {\text{bQhat}}^{\theta }(x;\mathbf{u})-\mathbf{r}(x;\mathbf{u})-{\gamma }_i\underset{{\mathbf{u}}^{\prime }\in \mathcal{U}}{N}{\text{bQhat}}^{\theta }(x^{\prime };{\mathbf{u}}^{\prime })\Vert }^2\right],$$

$$\mathcal{L}(\theta )=\frac{1}{M}\sum _{m=1}^M{\Vert {\text{bQhat}}^{\theta }(x_m;{\mathbf{u}}_m)-\mathbf{r}(x_m;{\mathbf{u}}_m)-{\gamma }_i\underset{{\mathbf{u}}_m^{\prime }\in \mathcal{U}}{N}{\text{bQhat}}^{\theta }(x_m^{\prime };{\mathbf{u}}_m^{\prime })\Vert }^2,$$

$${\text{bQhat}}^{\theta }(x;\mathbf{u})={\text{bVhat}}^{\theta }(x)+{\hat{\mathbf{A}}}^{\theta }(x;\mathbf{u}),$$

$${\hat{A}}_i^{\theta }(x;\mathbf{u})=$$

$${\mathbf{P}}_i^{\theta }(x):=\left(\begin{array}{cc}{P}_{11,i}^{\theta }(x)& P_{12,i}^{\theta }(x)\\ P_{21,i}^{\theta }(x)& P_{22,i}^{\theta }(x)\end{array}\right),$$

$${\text{bVhat}}^{\theta }(x)=\underset{u\in \mathcal{U}}{N}{\text{bQhat}}^{\theta }(x;\mathbf{u})\text{and}\mathbf{\mu }(x)=\arg \underset{u\in \mathcal{U}}{N}{\text{bQhat}}^{\theta }(x;\mathbf{u}).$$

$${\mathcal{L}}_m(\theta )={\Vert {\text{bVhat}}^{\theta }(x_m)+{\hat{\mathbf{A}}}^{\theta }(x_m;{\mathbf{u}}_m)-\mathbf{r}(x_m;{\mathbf{u}}_m)-{\gamma }_i{\text{bVhat}}^{\theta }(x_m^{\prime })\Vert }^2,$$

$$\mathcal{L}(\theta )=\frac{1}{M}\sum _{m=1}^M{\mathcal{L}}_m(\theta )$$

$$r_i\left(x_0,x_i,\lambda (x_{-i});u_i,\lambda (u_{-i})\right)=r_i\left(x_0,x_i,x_{-i};u_i,u_{-i}\right).$$

$${\hat{A}}_i^{\theta }(x;\mathbf{u})=$$

$$-\sum _{j\in \mathfrak{N}/\{i\}}\Vert u_j-{\mu }_j^{\theta }(x){\Vert }_{P_{22,i}^{\theta }(x)}^2+\sum _{j\in \mathfrak{N}/\{i\}}{\left(u_j-{\mu }_j^{\theta }(x)\right)}^{⊺}{\mathbf{\psi }}^{\theta }(x),$$

$$f_{\text{inv}}(z)=\sigma \left(\sum _{j=1}^J\varphi (z_j)\right).$$

$$\frac{1}{M}\sum _{m=1}^M\hat{\mathcal{L}}\left(y_m,{\theta }_V,{\theta }_A\right),$$

$$\hat{\mathcal{L}}(y_m,{\theta }_V,{\theta }_A)={\Vert {\text{bVhat}}^{{\theta }_V}(x_m)+{\hat{\mathbf{A}}}^{{\theta }_A}(x_m;{\mathbf{u}}_m)-\mathbf{r}(x_m;{\mathbf{u}}_m)-{\gamma }_i{\text{bVhat}}^{{\theta }_V}(x_m^{\prime })\Vert }^2,$$

$$d{S}_t$$

$$d{Y}_t$$

$$d{S}_t$$

$$d{Y}_t$$

$${\mathcal{R}}_i:=\mathbb{E}\left[X_{i,T}+q_{i,T}\left(S_T-b_2{q}_{i,T}\right)-b_3{\int }_0^T{q}_{i,t}^{2}dt\right],$$

$$\tilde{L}(y_m,{\theta }_V,{\theta }_A):=\hat{\mathcal{L}}(y_m,{\theta }_V,{\theta }_A)+\beta \Vert {\mathbf{\psi }}^{\theta }(x)\Vert$$

Full text

1 Introduction

The study of equilibria in systems of interacting agents is ubiquitous throughout the natural and social sciences. The classical approach to studying these equilibria requires building a model of the interacting system, solving for its equilibrium, and studying its properties thereafter. This approach often runs into complications, however, as a fine balance between (i) model tractability and (ii) its ability to capture the main features of the data it aims to represent, must be struck. Rather than taking a model-based approach, it is possible to derive non-parametric reinforcement-learning (RL) methods to study these equilibria. The main idea behind these methods is to directly approximate equilibria from simulations or observed data, providing a powerful alternative to the usual approach.

The majority of the existing literature on RL is dedicated to single-player games. Most modern approaches follow either a deep Q-learning approach(e.g. [22]), policy gradient methods (e.g. [26]), or some mixture thereof (e.g. [11]). RL methods have also been developed for multi-agent games, but are for the most part restricted to the case of zero-sum games. For a survey see [3].

There are recent efforts on extending RL to general sum games with fictitious play as in [13], or with iterative fixed point methods as in [18]. In the specific context of (discrete state-action space) mean-field games, [9] provides a Q-learning algorithm for solving for the Nash-equilibria. Many of the existing algorithms suffer either from computational intractability as the size and complexity of a game increases, when state-action space becomes continuous, or from the inability to model complex game behaviour.

Hu and Wellman [12] introduce a Q-learning based approach for obtaining Nash equilibria in general-sum stochastic games. Although they prove convergence of the algorithm for games with finite game and action spaces, their approach is computationally infeasible for all but the simplest examples. The main computational bottleneck in their approach is the need to repeatedly compute a local Nash equilibrium over states, which is an NP-hard operation in general. Moreover, the method proposed in [12] does not extend to games where agents choose continuous-valued controls or to games with either high-dimensional game state representations or with large numbers of players. We instead combine the iLQG framework of of [27, 8] and the Nash Q-learning algorithm of [12] to produce an algorithm which can learn Nash equilibria in these more complex and practically relevant settings.

In particular, we decompose the state-action value (Q)-function as a sum of the value function and the advantage function. We approximate the value function using a neural-net, and we locally approximate the advantage function as linear-quadratic in the agents’ actions with coefficients that are non-linear functions of the features given by a neural-net. This allows us to compute the Nash equilibrium analytically at each point in feature space (i.e., the optimal action of all agents) in terms of the network parameters. Using this closed form local Nash equilibrium, we derive an iterative actor-critic algorithm to learn the network parameters.

In principle, our approach allows us to deal with stochastic games with a large number of game state features and a large action space. Moreover, our approach can be easily adapted to mean-field game (MFG) problems, which result from the infinite population limit of certain stochastic games (see [15, 19, 4]), such as those developed in, e.g., [6, 5] or major-minor agent MFGs such as those studied in, e.g., [14, 25, 16]. A drawback of the method we propose is the restriction on the local structure of the proposed Q-function approximator. We find, however, that the proposed approximators are sufficiently expressive in most cases, and perform well in the numerical examples that we include in this paper.

The remainder of this paper is structured as follows. Section 2 introduces a generic Markov model for a general-sum stochastic game. In Section 3, we present optimality conditions for the stochastic game and motivate our Q-learning approach to finding Nash equilibria. Section 4 introduces our local linear-quadratic approximations to the Q-function and the resulting learning algorithm. We also provide several simplifications that arise in label-invariant games. Section 5 covers implementation details and Section 6 presents some illustrative examples.

2 Model Setup

We consider a stochastic game with agents $i\in{\mathfrak{N}}:=\{1,2,\dots,N\}$ all competing together. We assume the state of the game is represented via the stochastic process $\{x_{t}\}_{t\in{\mathds{N}}}$ so that for each time $t$, $x_{t}\in\mathcal{X}$, for some separable Banach space $\mathcal{X}$. At each time $t$, agent-$i$ chooses an action $u_{i,t}\in\mathcal{U}_{i}$, where $\mathcal{U}_{i}$ is assumed to be a separable Banach space. In the sequel, we use the notation ${\bm{u}}_{-i,t}=(u_{j,t})_{j\in{\mathfrak{N}}/\{i\}}$ to denote the vector of actions of all agents other than agent-$i$ at time $t$ and the notation ${\bm{u}}_{t}=(u_{j,t})_{j\in{\mathfrak{N}}}$ to denote the vector of actions of all agents. We assume that the game is a Markov Decision Process (MDP) with a fully-visible game state. The MDP assumption is equivalent to assuming the joint state-action process $\left(x_{t},{\bm{u}}_{t}\right)_{t=1}^{\infty}$ is Markov, whose state transition probabilities are defined by the stationary Markov transition kernel $p(x_{t+1}\mid x_{t},{\bm{u}}_{t})$ and a distribution over initial states $p_{0}(x_{0})$.

At each step of the game, agents receive a reward that varies according to the current state of the game, their own choice of actions, and the actions of all other agents. The agent-$i$’s reward is represented by the function $(x,u_{i},{\bm{u}}_{-i})\mapsto r_{i}(x;u_{i},{\bm{u}}_{-i})\in\mathds{R}$, so that at each time $t$, agent-$i$ accumulates a reward $r_{i}(x_{t};u_{i,t},{\bm{u}}_{-i,t})$. We assume that each function $r_{i}$ is continuously differentiable and concave in $u_{i}$ and is continuous in $x$ and ${\bm{u}}_{-i}$.

At each time $t$, agent-$i$ may observe other agents’ actions ${\bm{u}}_{-i,t}$, as well as the state of the game $x_{t}$. Moreover, each agent-$i$ chooses their actions according to a deterministic Markov policy $\mathcal{X}\ni x\mapsto\pi_{i}(x)\in\mathcal{U}_{i}$. The objective of agent-$i$ is to select the policy $\pi_{i}$ that maximizes the objective functional $\mathcal{R}_{i}$ which represents their personal expected discounted future reward over the remaining course of the game, given a fixed policy $\pi_{i}$ for themselves and a fixed policy $\bm{\pi}_{-i}$ for all other players. The objective functional for agent-$i$ is

[TABLE]

where the expectation is over the process $(x_{t})_{t\in{\mathds{N}}}$, with $x_{0}=x$, and where we assume $\gamma_{i}\in(0,1)$ is a fixed constant representing a discount rate. In Equation (1), we use the compressed notation $\pi_{i,t}:=\pi_{i}(x_{t})$ and $\bm{\pi}_{-i,t}:=\bm{\pi}_{-i}(x_{t})$. The agent’s objective functional (1) explicitly depends on the policy choice of all agents. Each agent, however, can only control their own policy, and must choose their actions while conditioning on the behavior of all other players.

Agent-$i$ therefore seeks a policy that optimizes their objective function, but remains robust to the actions of others. In the end, agents’ policies form a Nash equilibrium – a collection of policies $\bm{\pi}^{\ast}(x)=\{\bm{\pi}^{\ast}_{i}(x)\}_{i\in{\mathfrak{N}}}$ such that unilateral deviation from this equilibrium by a single agent will result in a decrease in the value of that agent’s objective functional. Formally, we say that a collection of policies $\bm{\pi}^{\ast}$ forms a Nash equilibrium if

[TABLE]

for all admissible policies $\pi_{i}$ and for all $i\in{\mathfrak{N}}$. Informally, we can interpret the Nash equilibrium as the policies for which each agent simultaneously maximizes their own objective function, while conditioned on the actions of others.

3 Optimality Conditions

Our ultimate goal is to obtain an algorithm that can attain the Nash equilibrium of the game without a-priori knowledge of its dynamics. In order to do this, we first identify conditions that are more easily verifiable than the formal definition of a Nash equilibrium given above.

We proceed by extending the well known Bellman equation for Nash equilibria. While leaving $\bm{\pi}^{\ast}_{-i}$ fixed, we may apply the dynamic programming principle to agents-$i$ reward $\mathcal{R}_{i}(x;\pi^{\ast}_{i},\bm{\pi}^{\ast}_{-i})$ resulting in

[TABLE]

At the Nash equilibrium, equation (3) is satisfied simultaneously for all $i\in{\mathfrak{N}}$.

To express this more concisely, we introduce a vector notation. First define the vector-valued function $\bm{\mathcal{R}}(x;\bm{\pi})=(\mathcal{R}_{i}(x;\pi_{i},\bm{\pi}_{-i}))_{i\in{\mathfrak{N}}}$, consisting of the stacked vector of objective functions. We call the stacked objective functions evaluated at their Nash equilibria the stacked value function, which we write as $\bm{V}(x):=\left(V_{i}(x)\right)_{i{\mathfrak{N}}}=\bm{\mathcal{R}}(x;\bm{\pi}^{\ast})$.

Next, we define the Nash state-action value function, also called the Q-Function, which we denote $\,\bm{Q}(x;{\bm{u}}):=\left(Q_{i}(x;u_{i},{\bm{u}}_{-i})\right)_{i\in{\mathfrak{N}}}$, where

[TABLE]

and where we denote $\bm{r}(x,{\bm{u}}):=\left(r_{i}(x,u_{i},{\bm{u}}_{-i})\right)_{i\in{\mathfrak{N}}}$ to indicate the vectorized reward function. Each element $Q_{i}$ of $\bm{Q}$ can be interpreted as the expected maximum value their objective function may take, given a fixed current state $x$ and a fixed (arbitrary) immediate action ${\bm{u}}$ taken by all agents.

Next, we define the Nash operator as follows.

Definition 3.1** (Nash Operator).**

Consider a collection of $N$ concave real-valued functions, $\bm{f}({\bm{u}})=(f_{i}(u_{i},{\bm{u}}_{-i}))_{i\in{\mathfrak{N}}}$, where $f_{i}:\mathcal{U}_{i}\bigtimes\mathcal{U}_{-i}\rightarrow\mathds{R}$. We define the Nash operator $\mathchoice{\underset{{\bm{u}}\in\bm{\mathcal{U}}}{\bm{\mathcal{N}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\bm{\mathcal{U}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\bm{\mathcal{U}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\bm{\mathcal{U}}}}\bm{f}({\bm{u}})\mapsto\bm{f}({\bm{u}}^{\ast})$, as a map from the collection of functions to their Nash equilibrium value ${\bm{u}}^{\ast}\in\mathds{R}^{N}$, where ${\bm{u}}^{\ast}=\arg\mathchoice{\underset{{\bm{u}}\in\bm{\mathcal{U}}}{\bm{\mathcal{N}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\bm{\mathcal{U}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\bm{\mathcal{U}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\bm{\mathcal{U}}}}\bm{f}({\bm{u}})$ is the unique point satisfying,

[TABLE]

For a sufficiently regular collection of functions $\bm{f}$, the Nash operator corresponds to simultaneously maximizing each of the $f_{i}$ in their first argument $u_{i}$.

This definition provides us with a relationship between the value function and agents’ $Q$-function as $\bm{V}(x)=\mathchoice{\underset{{\bm{u}}\in\bm{\mathcal{U}}}{\bm{\mathcal{N}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\bm{\mathcal{U}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\bm{\mathcal{U}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\bm{\mathcal{U}}}}\bm{Q}(x;{\bm{u}})$. Using the Nash operator, we may then express the Bellman Equation (3) in a concise form as

[TABLE]

which we refer to as the Nash-Bellman equation for the remainder of the paper. The definition of the value function equation (6) implies that $\bm{\pi}^{\ast}=\mathchoice{\underset{{\bm{u}}\in\bm{\mathcal{U}}}{\bm{\mathcal{N}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\bm{\mathcal{U}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\bm{\mathcal{U}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\bm{\mathcal{U}}}}\bm{Q}(x;{\bm{u}})$. Hence, in order to identify the Nash equilibrium $\bm{\pi}^{\ast}$, it is sufficient to obtain the $Q$-function and apply the Nash operator to it. This principle will inform the approach we take in the remainder of the paper: rather than directly searching the space of policy collections for the Nash-equilibrium via equations (1) and (2), we may rely on identifying the function $\bm{Q}$ satisfying (6), and thereafter compute $\bm{\pi}^{\ast}=\mathchoice{\underset{{\bm{u}}\in\bm{\mathcal{U}}}{\bm{\mathcal{N}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\bm{\mathcal{U}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\bm{\mathcal{U}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\bm{\mathcal{U}}}}\bm{Q}(x;{\bm{u}})$.

4 Locally Linear-Quadratic Nash Q-Learning

In this section, we formulate an algorithm which learns the Nash equilibrium of the stochastic game described in the previous section. The principal idea behind the approach we take is to construct a parametric estimator $\bQhat^{\theta}$ of agent-$i$’s $Q$-function, where we search for the set of parameters $\theta\in\Theta$, which results in estimators $\bQhat^{\theta}$ that approximately satisfy the Nash-Bellman equation (6). Thus, our objective is to minimize the quantity

[TABLE]

over all $u$, where we define $\rho$ to be an unconditional proability measure over game states $x$. Equation (7) is designed as a measure of the gap between the right and left sides of equation (6). We may also interpret it as the distance between $\bQhat^{\theta}$ and the true value of $\bm{Q}$. The expression (7) is intractable, since we do not know $\rho$ nor $p(\cdot|x,{\bm{u}})$ a-priori, and we wish to make little to no assumptions on the system dynamics. Therefore, we rely on a simulation based method and approximate (7) with

[TABLE]

where for each $m=1,2,\dots,M$, $(x_{m},{\bm{u}}_{m};x^{\prime}_{m})$ represents an observed transition triplet from the game. We then search for $\theta\in\Theta$ that minimizes the $\mathcal{L}(\theta)$ in order to approximate $\bQhat$.

Our approach is motivated by Hu and Wellman [12] and Todorov & Li [27]. [12] presents a $Q$-learning algorithm where $\bQhat$, which is assumed to take only finitely many values, can be estimated through an update rule that relies on the repeated computation of the Nash operator $\mathchoice{\underset{{\bm{u}}\in\mathcal{U}}{\bm{\mathcal{N}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\mathcal{U}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\mathcal{U}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\mathcal{U}}}\bQhat^{\theta}$. As the computation of $\mathchoice{\underset{{\bm{u}}\in\mathcal{U}}{\bm{\mathcal{N}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\mathcal{U}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\mathcal{U}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}{\bm{u}}\in\mathcal{U}}}\bQhat^{\theta}$ is NP-hard in general, this approach proves to be computationally intractable beyond trivial examples. To circumvent this issue and to make use of more expressive parametric models, we generalize and adapt techniques in Gu et al. [8] to the multi-agent game setting to develop a computational and data efficient algorithm for approximating the Nash equilibria.

In our algorithm, we make the additional assumption that game states $x\in\mathcal{X}$ and actions $u_{i}\in\mathcal{U}_{i}$ are real-valued. Specifically, we assume that $\mathcal{X}=\mathds{R}^{d_{x}}$ for some positive integer $d_{x}$ and $\mathcal{U}_{i}=\mathds{R}^{d_{i}}$ for each $i\in{\mathfrak{N}}$, where $d_{1},\dots d_{N}$ are all positive integers. For notational convenience we define $d_{-i}:=\sum_{j\in{\mathfrak{N}}/\{i\}}d_{j}$.111Our approach can be easily extended to the case of controls that are restricted to convex subsets of $\mathds{R}^{d_{i}}$.

We now define a specific model for the collection of approximate $Q$-functions $\bQhat^{\theta}(x;{\bm{u}})=(\hat{Q}^{\theta}_{i}(x;{\bm{u}}))_{i\in{\mathfrak{N}}}$. For each $\theta\in\Theta$, we have $\bQhat^{\theta}:\mathcal{X}\bigtimes\bm{\mathcal{U}}\rightarrow\mathds{R}^{N}$ and decompose the $Q$-function into two components:

[TABLE]

where $\bVhat^{\theta}(x)=(\hat{V}^{\theta}_{i}(x))_{i\in{\mathfrak{N}}}$ is a model of the collection of value functions so that $\bVhat^{\theta}:\mathcal{X}\rightarrow\mathds{R}^{N}$ and where $\bm{\widehat{A}}^{\theta}(x;{\bm{u}})=(\widehat{A}_{i}^{\theta}(x;{\bm{u}}))_{i\in{\mathfrak{N}}}$ is what we refer to as the collection of advantage functions. The advantage function represents the optimality gap between $\bQhat$ and $\bVhat$. We further assume that for each $i\in{\mathfrak{N}}$, $\widehat{A}^{\theta}_{i}$ has the linear quadratic form

[TABLE]

where the block matrix

[TABLE]

$\mu^{\theta}_{i}(x):\mathcal{X}\rightarrow\mathds{R}^{d_{i}}$ with $\bm{\mu}^{\theta}_{-i}(x)=(\mu^{\theta}_{i}(x))_{i\in{\mathfrak{N}}}$, and $\bm{\Psi}_{i}:\mathcal{X}\rightarrow\mathds{R}^{d_{-i}}$. In (11), $P^{\theta}_{11,i}:\mathcal{X}\rightarrow\mathds{R}^{d_{i}\times d_{i}}$, $P^{\theta}_{12,i}:\mathcal{X}\rightarrow\mathds{R}^{d_{i}\times d_{-i}}$, $P^{\theta}_{21,i}:\mathcal{X}\rightarrow\mathds{R}^{d_{-i}\times d_{i}}$ and $P^{\theta}_{22,i}:\mathcal{X}\rightarrow\mathds{R}^{d_{-i}\times d_{-i}}$ are matrix valued functions, for each $i\in{\mathfrak{N}}$. We require that $P^{\theta}_{11,i}(x)$ is positive-definite for all $x\in\mathcal{X}$ and without loss of generality we may choose $P_{12,i}(x)=\left(P_{21,i}(x)\right)^{\intercal}$, as the advantage function depends only the symmetric combination of $P_{12,i}$ and $P_{21,i}$.

Hence, rather than modelling $\bQhat^{\theta}(x;{\bm{u}})$, we instead model the functions $\bVhat^{\theta}$, $\bm{\mu}^{\theta}$, and $\{\bm{P}^{\theta}_{i},\bm{\Psi}_{i}^{\theta}\}_{i\in{\mathfrak{N}}}$ separately as functions of the state space $\mathcal{X}$. Each of these functions can be modeled by universal function approximators such as neural networks. The only major restriction is that $P_{11,i}^{\theta}(x)$ must remain a positive-definite function of $x$. This restriction is easily attained by decomposing $P_{11,i}^{\theta}(x)$ using Cholesky decomposition, so that we write $P_{11,i}^{\theta}(x)=L_{11,i}^{\theta}(x)(L_{11,i}^{\theta}(x))^{\intercal}$ and instead model the lower triangular matrices $L_{11,i}^{\theta}\in\mathds{R}^{d_{i}\times d_{i}}$.

The model assumption in (10) implicitly assumes that agent-$i$’s $Q$-function can be approximately written as a linear-quadratic function of the actions of each agent. One can equivalently motivate such an approximation by considering a second order Taylor expansion of $Q_{i}$ in the variable ${\bm{u}}$ around the Nash equilibrium, together with the assumption that the $Q_{i}$ are convex functions of their input $u_{i}$. This expansion, however, assumes nothing about the dependence of $Q_{i}$ on the value of game state $x$.

The form of (10) is designed so that each $\hat{Q}_{i}^{\theta}(x;u_{i},{\bm{u}}_{-i})$ is a concave function of $u_{i}$, guaranteeing that $\mathchoice{\underset{u\in\bm{\mathcal{U}}}{\bm{\mathcal{N}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}u\in\bm{\mathcal{U}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}u\in\bm{\mathcal{U}}}}{\bm{\mathcal{N}}_{\vspace{1em}\scriptscriptstyle{}u\in\bm{\mathcal{U}}}}\bQhat$ is bijective. Moreover, under our model assumption, the Nash-equilibrium is attained at the point $u^{*}(x)=\bm{\mu}(x)$ and at this point, the advantage function is zero, hence we obtain simple expressions for the value function and the equilibrium strategy

[TABLE]

Consequently, our model allows us to directly specify the Nash equilibrium strategy and the value function of each agent through the functions $\bm{\mu}^{\theta}$ and $\bVhat^{\theta}$. The outcome of this simplification is that the summand of the loss function in equation (8), which contains the Nash equilibria and was itself previously intractable, becomes tractable. For each sample observation $m$ (consisting of a state $x_{m}$, an ${\bm{u}}_{m}$, and new state $x^{\prime}_{m}$) we then have a loss of

[TABLE]

over the parameters $\theta$ given a set of observed state-action triples $(x_{m},{\bm{u}}_{m},x^{\prime}_{m})_{m=1}^{M}$.

4.1 Simplifying Game Structures

Equation (10) requires a parametric model of the functions $\bVhat^{\theta}$, $\bm{\mu}^{\theta}$,$\{\bm{P}^{\theta}_{i},\bm{\Psi}_{i}\}_{i\in{\mathfrak{N}}}$, which results in potentially a very large parameter space and in principle result in requiring many training steps. In many cases, however, the structure of the game can significantly reduce the dimension of the parameter space and leads to easily learnable model structures. The following subsections enumerate these typical simiplications.

Label Invariance

Many games have symmetric players, and hence are invariant to a permutation of the label of players. Such label invariance implies that each agent-$i$ does not differentiate among other game participants and the agent’s reward functional is independent of any reordering of all other agents’ states and/or actions.

More formally, we assume that for an arbitrary agent-$i$, the game’s state can be represented as $x=(x_{0},x_{i},x_{-i})$, where $x_{0}$ represents the part of the game state not belonging to any agent, $x_{i}$ represents the portion of the game state belonging to agent-$i$ and $x_{-i}=\{x_{j}\}_{j\in{\mathfrak{N}}/\{i\}}$ represents the part of the game state belonging to other agents. Next, let $\Lambda$ denote the set of permutations over sets of $N-1$ indices, where for each $\lambda\in\Lambda$, we express the permutation of a collection as $\lambda(\{y_{j}\}_{j=1}^{N-1})=\{y_{k(j)}\}_{j=1}^{N-1}$, where $k:\{1,\dots,N-1\}\rightarrow\{1,\dots,N-1\}$ is a one-to-one and onto map from the indices of the collection into itself.

Label invariance is equivalent to the assumption that for any $\lambda\in\Lambda$, each agent’s reward function satisfies

[TABLE]

With such label invariance, the form of the linear quadratic expansion of the advantage function in (10) simplifies. Assuming that $d_{j}=d$, for all $j\in{\mathfrak{N}}$, independent label invariance in only the actions of agents requires $\widehat{A}_{i}$ to have the simplified form

[TABLE]

for all $i\in{\mathfrak{N}}$, where we use the notation $\|z\|^{2}_{M}=z^{\intercal}Mz$ and $\langle y,z\rangle_{M}=y^{\intercal}Mz$ for appropriately sized matrices $M$. The functional form of (15) allows us to drastically reduce the size of the matrices being modelled by an order of $N^{2}$.

To impose label invariance on states, we require permutation invariance on the inputs the function approximations $\bVhat^{\theta}$, $\bm{\mu}^{\theta}$, $\{\bm{P}_{i},,\bm{\psi}^{\theta}\}_{i\in{\mathfrak{N}}}$. [28] provide necessary and sufficient conditions on neural network structures to be permutation invariant. This necessary and sufficient structure is defined as follows. Let $\phi:\mathds{R}^{n_{1}}\rightarrow\mathds{R}^{n_{2}}$ and $\sigma:\mathds{R}^{n_{2}}\rightarrow\mathds{R}^{n_{3}}$ be two arbitrary functions. From these functions, let $f_{inv}:\mathds{R}^{J\times n_{1}}\rightarrow\mathds{R}^{n_{3}}$ be the composition of these functions, such that

[TABLE]

It is clear that $f_{inv}$ constructed in this manner is invariant to the reordering of the components of $z$. Equation (16) may be interpreted as a layer which aggregates the all dimensions of the inputs (which will corresponding to the state of all agents), through $\phi$, and a layer that transforms the aggregate result to the output, through $\sigma$. We assume further that $\phi$ and $\sigma$ are both neural networks with appropriate input and output dimension. This structure can also be embedded as an input later inside of a more complex neural network.

Identical Preferences

It is quite common that the admissible actions of all agents are identical, i.e., $\mathcal{U}_{i}=\mathcal{U}$, $\forall i\in{\mathfrak{N}}$, and agents have homogeneous objectives, or large sub-populations of agents have homogeneous objectives. Thus far, we allowed agents to assign different performance metrics, and the variations are show through the set of rewards and discount rates, $\{r_{i},\gamma_{i}\}_{i\in{\mathfrak{N}}}$. If agents have identical preferences, then we simply need to assume $r_{i}(x;{\bm{u}})=r(x;{\bm{u}})$ and $\gamma_{i}=\gamma$ for all $i\in{\mathfrak{N}}$. By the definition of total discounted reward, state-action value function, and value function, identical preferences and admissible actions imply that $\mathcal{R}_{i}$, $Q_{i}$ and $V_{i}$ are independent of $i$.

In addition, the assumption of identical preferences, combined with the assumption of label invariance can further reduce the parametrization of the advantage function. Under this additional assumption we have that $\hat{V}_{i},\widehat{A}_{i}$ must be identical for all $i$, which reduces modelling of all of the $\hat{V}_{i}^{\theta}$, $\mu_{i}^{\theta}$,$\bm{P}_{i}^{\theta}$,$\bm{\psi}^{\theta}$ to modelling these for a single $i$. This further reduces the number of functions that must be modeled by an order of $N$. The combined effect of label invariance and identical preferences has a compounding effect which can have a large impact on the modelling task, particularily when considering large populations of players.

Remark 4.1** (Sub-population Invariance and Preferences).**

*We can also consider cases where label and preference invariance occur within sub-population of agents, rather than across the entire population. For example, in games in which some agents may cooperate with other agents, we can assume that agents are indifferent to re-labeling of cooperators and non-cooperators separately. Similarly, we can consider cases in which groups of agents share the same performance metrics. Such situations, among others, lead to modelling simplifications similar to equation (15) and simplifying neural network structures can be developed. In the interest of space, we do not develop further examples simplifying examples, nor do we claim the list we provide is exhaustive as one can easily imagine a multitude of other almost symmetric cases that can be of interest. *

5 Implementation of Nash Actor-Critic Algorithm

With the locally linear-quadratic form of the advantage function, and the simplifying assumptions outlined in the previous section, we can now minimize the objective (8), which reduces to the sum over (13b), over the parameters $\theta$ through an iterative optimization and sampling scheme. One could in principle apply a simple stochastic gradient descent method using back-propagation on the appropriate loss function. Instead, we propose an actor-critic style algorithm to increase stability and efficiency of the algorithm. Actor-critic methods (see e.g. [17]) have been shown to provide faster and more stable convergence of reinforcement learning methods towards their optima, and our model lends itself naturally to such methods.

The decomposition in Equation (9) allows us to model the value function $\hat{V}$ independently from other components. Therefore, we employ an actor-critic update rule to minimize the loss function (13b) by separating the parameter set $\theta=(\theta_{V},\theta_{A})$, where $\theta_{V}$ represents the parameter set for modelling $\hat{V}^{\theta_{V}}$ and $\theta_{A}$ represents the parameter set used for modeling $\bm{\widehat{A}}^{\theta_{A}}$. Our proposed actor-critic algorithm updates these parameters by minimizing the total loss

[TABLE]

with ${\bm{u}}_{m}$, $y_{m}=\left(x_{m},{\bm{u}}_{m},x_{m}^{\prime}\right)$, and we minimize the loss by alternating between minimization in the variables $\theta_{A}$ and $\theta_{V}$.

Algorithm 1 provides an outline of the actor-critic procedure for our optimization problem. We include a replay buffer and employ mini-batching. A replay buffer is a collection of previously experienced transition tuples of the form $y_{t}=(x_{t-1},{\bm{u}},x_{t})$ representing the previous state of the system, the action taken in that state, the resulting state of the system, and the reward during the transition. We randomly sample a mini-batch from the replay buffer to update the model parameters using SGD. The algorithm also uses a naïve Gaussian exploration policy, although it may be replaced by any other action space exploration method. During the optimization steps over $\theta_{V}$ and $\theta_{A}$, we use stochastic gradient descent, or any other adaptive optimization methods.

6 Experiments

We test our algorithm on a multi-agent game for statistical arbitrage in electronic exchanges. The game consists of agents trading a single asset with a stochastic price process that is affected by their actions. A simpler version of this model has been studied in [5, 6, 24]. In these works, under various assumptions, the authors take a mean-field approach and show that the mean field optimal strategy provides an approximate Nash equilibrium for the finite-player case.

In our setting, an arbitrary agent-$i$, $i\in{\mathfrak{N}}$, may buy or sell assets at a rate of $\nu_{i,t}\in\mathds{R}$ during the trading horizon $t\in\mathfrak{T}:=[0,T]$. At $t=T$, agents must liquidate their holdings, any remaining inventory will be subjected to a terminal penalty. Agents may change their rate of trading at discrete specific points in time $t_{1},t_{2},\dots,t_{M}$ (decision points). The rate of trading is assumed to be constant between any two decision points and thus the total amount of assets traded during the period $(t_{m},t_{m+1})$ is given by $\nu_{i,t_{m}}^{T}=(t_{m+1}-t_{m})\nu_{i,t_{0}}$. Each agent-$i$ keeps track of their inventory $q_{i,t_{m}}=q_{i,0}+\int_{0}^{t_{m}}\nu_{i,t}\,dt$. Inventories are not visible to other agents however the total order flow, i.e. $\sum_{i\in{\mathfrak{N}}}(q_{i,t}-q_{i,0})$, is. For ease of notation, we assume $t\in\{t_{1},t_{2},\dots,t_{M}\}$. We assume the asset price process $S_{t}$ evolves according to the following system of continuous time dynamics:

[TABLE]

with initial condition $S_{0}$, variance $\sigma>0$, and Brownian motion $(W_{t})_{t\geq 0}$. Furthermore, we denote the vector valued process of trading rates by $\bm{\nu}=(\left(\nu_{i,t}\right)_{i\in{\mathfrak{N}}})_{t\geq 0}$ and denote by $\bar{\nu}_{t}=\sum_{i\in{\mathfrak{N}}}\nu_{i,t}$ the total trading rate at time $t$. The above form assumes the cumulative trades of all agents induce both a transient impact through $Y:=(Y_{t})_{t\geq 0}$ and a permanent impact through $g$.

The functions $\mu$ and $g$ represents the mean process and permanent price impact respectively. For the experiments conducted here, we assume the process mean-reverts so that $\mu(S)=\kappa(\theta-S)$, where $\kappa>0$ represents the mean-reversion rate and $\theta$ the mean-reversion level. Moreover, we assume a linear permanent price impact so that $g(\bm{\nu})=\eta\,\bar{\nu}$. The cumulative transient price impact is assumed to be square root, i.e. $h(\bm{\nu})=sign(\bar{\nu})\sqrt{\bar{\nu}}$. Under these model assumptions, the continuous dynamics may be written

[TABLE]

In addition, each agent pays a transaction cost proportional to the amount they buy or sell during each time period. This represents the cost incurred by the agents from “walking the limit-order-book” and is often dependent on the liquidity of the asset. Agents keep track of their total cash from trading and we denote the corresponding process by $X_{i}:=(X_{i,t})_{t\geq 0}$ where $X_{i,t}=-\int_{0}^{T}\nu_{i,t}\left(S_{t}+b_{1}\nu_{i,t}\right)\,dt$, and $b_{1}>0$ is the transaction cost constant.

The agent’s objective is to maximize the sum of (i) total cash they possess by time $T$, (ii) excess exposure at time $T$, and (iii) a penalty for risk taking. We express agent-$i$’s objective (total expected reward) as

[TABLE]

where $b_{2},b_{3}>0$. In Equation (22), the second term serves as a cost of instantaneously liquidating the inventory at time $T$ and the last term serves as a penalty for taking on excess risk proportional to the square of the holdings at each time period – the so called urgency penalty. In this objective function, the effect of all agent’s trading actions appears implicitly through the dynamics of $S$, and through its effect on the cash process $X_{i}$. This particular form of objective assumes that agents have identical preferences which are invariant to agent relabeling222We could extend this to include the case of sub-populations with homogeneous preferences, but that are heterogeneous across sub-populations.. Hence, we may employ the techniques discussed in Section 4.1 to simplify the form of the advantage function $\widehat{A}$. In our example, we model each component of the advantage function $\widehat{A}$ with neural networks that includes a permutation invariant layer.

Our experiments assume a total of five agents over a time horizon of five hours ($T=5$) and ten decision points ($M=10$) of equal duration.

6.1 Features

We use the following features to represent the state $x_{t}$ of the environment at time $t$:

**Price ($S_{t}$): ** Scalar representing the current price of the asset,

**Time ($t$): ** Scalar representing the current time step the agent is at in the time horizon,

**Total order flow ($\sum_{i\in{\mathfrak{N}}}q_{i,0}-q_{i,t}$): ** Scalar representing the total order from all agents since the beginning of the trading horizon, and

**Cumulative transient price impact ($Y_{t}$): ** Scalar representing the current level of transient price impact effecting the price.

All features are assumed to be non-label invariant.

6.2 Network Details

The network structure for the advantage function approximation $\bm{\widehat{A}}^{\theta_{A}}$ consists of two network components: (i) a permutation invariant layer that feeds into (ii) a main network layer. The input of the permutation invariant layer are the label invariant features. This layer, as described in Section 4.1, is a fully connected neural network with three hidden layers each containing 20 nodes. Layers are connected by SiLU activation functions [7]. We then combine the output of this permutation invariant later with the non-label invariant features and together they form the inputs to the main network. The main network comprises of four hidden layers with 32 nodes each. The outputs of this main network are the parameters $\bm{\mu}^{\theta}$ and $\{\bm{P}^{\theta}_{i},\bm{\Psi}_{i}\}_{i\in{\mathfrak{N}}}$, of the approximated advantage function defined in Section 4. These parameters fully specify the value of the advantage function.

The network structure for the value function approximation $\hat{V}^{\theta_{V}}$ contains four hidden layers with 32 nodes each. This network takes the features from all states described in Section 6.1 and outputs the approximate value function for all agents.

We use the Adam optimizer [21] with mini-batches and a weight decay of 0.001 to optimize the loss functions defined in Section 5. Mini-batch sizes are set to ten full episodes executed on the period [0, T]. Learning rates are set to 0.003 and are held constant throughout training, modified only by the optimizer’s weight decay. Training is performed over a maximum of 20,000 iterations, with a early stopping criteria of no improvement in the loss in the last 3,000 iterations.

6.3 Baseline - Fictitious Play

Fictitious Play (FP), first introduced in [2], is a classical method of determining the Nash Equilibrium of multi-agent games. It has been shown to converge in the two player case for zero-sum games [2], potential games [23], $2\times N$ games with generic payoffs [1], and is often used as a basis for many modern methods [10]. In general, FP assumes each player follows a stationary mixed strategy that is updated at each iteration by taking the best response to the empirical average of the opponents’ previous strategies. In our experiments we assume identical agents and thus computing the optimal response for a single agent at each iteration is sufficient. Specifically, let ${{\bm{u}}^{F}_{i}(x)}_{i\in{\mathfrak{N}}}$ be the optimal FP strategies for each agent which we use the algorithm defined in Algorithm 2 to obtain.

To obtain the optimal response of agent-$i$ against other agents’ strategies in line 2 of Algorithm 2 we use Deep Deterministic Policy Gradient (DDPG) [20]. DDPG uses a neural network to represent the optimal policy and the state-action value function. As with our Nash-DQN approach, this approach has no knowledge of the agent’s reward function nor the transition dynamics, and hence provides a fair comparison for our Nash-DQN. Due, however, to the nature of DDPG combined with FP it is highly data and training inefficient as each FP iteration requires the optimization of the DDPG policy – and each iteration of DDPG uses approximately the same amount of resources as a full training cycle of our Nash-DQN method.

6.4 Optimization Improvements

We find that directly minimizing Equation 17b sometimes produce inaccuracies in the final policy due to a difference in magnitudes of the learned parameters used in the advantage function Equation 15, specifically the linear term $\bm{\psi}^{\theta}(x)$. Adding in the $L^{1}$ regularization term

[TABLE]

improves the performance significantly.

The optimal value for the hyperparameter $\beta$ is determined by minimizing the average loss Equation 17b computed over 1,000 randomly generated simulations. Figure 1 indicates an value of $\beta=100$ is optimal.

6.5 Results

We use the set of model parameters shown in Table 1 in our analysis.

Figure 2 shows the optimal policies obtained through our Nash-DQN method and the FP method. We can see that despite being significantly more data efficient, the optimal policies are essentially the same. To evaluate the performance of the Nash-DQN method, we considering the following two scenarios: (i) Agent 1 following Nash-DQN and all other agents FP vs. all agents following FP, (ii) Agent 1 following FP and all other agents Nash-DQN vs all agents following Nash-DQN. Next, we apply these policies to 1,000 paths using the simulated environment. Finally, we average total rewards, repeat the whole exercise 100 times, and plot the resulting distributions shown in Figure 3. From the figures, it appears there is no noticeable discernable difference in the distributions. Indeed, the null hypothesis that the means of the results from FP and Nash-DQN differ cannot be rejected at the 5% level.

7 Conclusions

Here we present a computationally tractable reinforcement leanring framework for multi-agent (stochastic) games. Our approach utilizes function approximations after decomposing the collection of agents’ state-action value functions into the individual value functions and their advantage functions. Further, we approximate the advantage function in a locally linear-quadratic form and use neural-net architectures to approximate both the value and advantage function. Typical symmetries in games allow us to use permutation invariant neural-nets, motivated by the Arnold-Kolmogorov representation theorem, to reduce the dimensionality of the parameter space. Finally, we develop an actor-critic paradigm to estimate parameters and apply our approach to an important application in electronic trading. Our approach is more data efficient than conventional FP policies, and is applicable to large number of players and continuous state-action spaces.

There are a number of doors left open for exploration including extending our approach to account for latent factors driving the environment, and when the state of all agents are partially (or completely) hidden from any individual agent. As well, our approach can be easily applied to mean-field games which correspond to the infinite population limit of stochastic games that have interactions where any individual agent has only an infinitesimal contribution to the state dynamics.