Computing Optimal Coarse Correlated Equilibria in Sequential Games

TL;DR

This paper introduces a new class of equilibria for extensive-form games, showing that optimal solutions can be computed efficiently in two-player cases but are NP-hard in more complex scenarios.

Contribution

It extends the concept of coarse correlated equilibrium to sequential games and provides polynomial-time algorithms for two-player cases, addressing computational complexity.

Findings

Optimal equilibria computed in polynomial time for two-player games without chance.

NP-hardness results for multi-player and games with chance.

Practical algorithms proposed for large two-player games.

Abstract

We investigate the computation of equilibria in extensive-form games where ex ante correlation is possible, focusing on correlated equilibria requiring the least amount of communication between the players and the mediator. Motivated by the hardness results on the computation of normal-form correlated equilibria, we introduce the notion of normal-form coarse correlated equilibrium, extending the definition of coarse correlated equilibrium to sequential games. We show that, in two-player games without chance moves, an optimal (e.g., social welfare maximizing) normal-form coarse correlated equilibrium can be computed in polynomial time, and that in general multi-player games (including two-player games with Chance), the problem is NP-hard. For the former case, we provide a polynomial-time algorithm based on the ellipsoid method and also propose a more practical one, which can be efficiently applied to problems of considerable size. Then, we discuss how our algorithm can be extended to games with Chance and games with more than two players.

Equations

$$\sum _{p_{-i}\in P_{-i}}{\sigma }^{\ast }(p_i,p_{-i})\left(U_i^{\prime }(p_i,p_{-i})-U_i^{\prime }(p_i^{\prime },p_{-i})\right)\ge 0.$$

$$\sum _{p_i\in P_i}\sum _{p_{-i}\in P_{-i}}{\sigma }^{\ast }(p_i,p_{-i})\left(U_i^{\prime }(p_i,p_{-i})-U_i^{\prime }(p_i^{\prime },p_{-i})\right)\ge 0.$$

$$\begin{gathered} \sum _{p_1\in P_1}\sum _{p_2\in P_2}\sigma (p_1,p_2)U_1^{\prime }(p_1,p_2)- \\ \sum _{p_1\in P_1}\sum _{p_2\in P_2}\sigma (p_1,p_2)U_1^{\prime }(p_1^{\prime },p_2)\ge 0\forall p_1^{\prime }\in P_1. \end{gathered}$$

$$\{\begin{array}{l}\sum _{p_1\in P_1}\sum _{p_2\in P_2}\sigma (p_1,p_2)U_1^{\prime }(p_1,p_2)=f_1^{\mathsf{T}}{v}_1\\ f_1^{\mathsf{T}}{v}_1-\sum _{p_1\in P_1}\sum _{p_2\in P_2}\sigma (p_1,p_2)U_1^{\prime }(p_1^{\prime },p_2)\ge 0\forall p_1^{\prime }\in P_1\end{array}.$$

$$\sum _{p_2\in P_2}\left(\sum _{p_1\in P_1}\sigma (p_1,p_2)\right)U_1^{\prime }(p_1^{\prime },p_2).$$

$$\underset{\sigma \ge 0,v_1,v_2}{\max }\sum _{(p_1,p_2)\in P_1\times P_2}\sigma (p_1,p_2)r_{p_1}^{\mathsf{T}}(U_1+U_2)r_{p_2}$$

$$\sum _{(p_1,p_2)\in P_1\times P_2}\sigma (p_1,p_2)r_{p_1}^{\mathsf{T}}{U}_i{r}_{p_2}=f_i^{\mathsf{T}}{v}_i\forall i\in N$$

$$\displaystyle F_{1}^{\mathsf{T}}v_{1}-U_{1}\bigg{(}\sum_{p_{2}\in P_{2}}\bigg{(}\sum_{p_{1}\in P_{1}}\sigma(p_{1},p_{2})\bigg{)}r_{p_{2}}\bigg{)}\geq 0$$

$$\displaystyle F_{2}^{\mathsf{T}}v_{2}-U_{2}^{\mathsf{T}}\,\bigg{(}\sum_{p_{1}\in P_{1}}\bigg{(}\sum_{p_{2}\in P_{2}}\sigma(p_{1},p_{2})\bigg{)}r_{p_{1}}\bigg{)}\geq 0$$

$$\sum _{(p_1,p_2)\in P_1\times P_2}\sigma (p_1,p_2)=1.$$

$$\begin{gathered} r_{p_1}^{\mathsf{T}}{U}_1{r}_{p_2}{\bar{\alpha }}_1+r_{p_1}^{\mathsf{T}}{U}_2{r}_{p_2}{\bar{\alpha }}_2+{\bar{\beta }}_1^{\mathsf{T}}{U}_1{r}_{p_2}+r_{p_1}^{\mathsf{T}}{U}_2{\bar{\beta }}_2+\bar{\gamma }\ge \\ {r}_{p_1}^{\mathsf{T}}(U_1+U_2)r_{p_2}. \end{gathered}$$

$$\min_{(p_{1},p_{2})\in P_{1}\times P_{2}}\Bigg{\{}r^{\mathsf{T}}_{p_{1}}\left((\bar{\alpha}_{1}-1)\,U_{1}+(\bar{\alpha}_{2}-1)\,U_{2}\right)\,r_{p_{2}}+\\ +\bar{\beta}_{1}^{\mathsf{T}}\,U_{1}\,r_{p_{2}}+r_{p_{1}}^{\mathsf{T}}\,U_{2}\,\bar{\beta}_{2}\Bigg{\}}.$$

$$x^{\mathsf{T}}=(\underset{|P_1\times P_2|}{\underbrace{\sigma (p_1^{\prime },p_2^{\prime }),\dots ,\sigma (p_1^{\prime \prime },p_2^{\prime }),\dots }},v_1^{\mathsf{T}},v_2^{\mathsf{T}},s_1^{\mathsf{T}},s_2^{\mathsf{T}}),$$

$$c^{\mathsf{T}}=(\underset{\sigma (p_1^{\prime },p_2^{\prime })}{\underbrace{[U_1^{\prime }(p_1^{\prime },p_2^{\prime })+U_2^{\prime }(p_1^{\prime },p_2^{\prime })]}},\dots ,\underset{|H_1|+|H_2|+|Q_1|+|Q_2|}{\underbrace{0,\dots ,0}}),$$

$${\overline{c}}_j=c_j-c_B^{\mathsf{T}}{B}^{-1}{M}_{(\cdot ,j)},$$

$$\underset{\begin{array}{c}z\in \{0,1{\}}^{|L|}\\ r_i\in {\mathbb{R}}_{+}^n\end{array}}{\max }$$

$$F_i{r}_i=f_i\forall i\in N$$

$$r_i\ge R_{i}z\forall i\in N$$

$$\sum _{\ell \in L}z(\ell )=1.$$

$$\begin{array}{r}{r}_1^{\mathsf{T}}\left(\left(1-{\bar{\alpha }}_1\right)U_{1,r_3}+\left(1-{\bar{\alpha }}_2\right)U_{2,r_3}+\left(1-{\bar{\alpha }}_3\right)U_{3,r_3}\right)r_2\\ -{\bar{\beta }}_1^{\mathsf{T}}{U}_{1,r_3}{r}_2-{\bar{\beta }}_2^{\mathsf{T}}{U}_{2,r_3}^{\mathsf{T}}{r}_1-{\bar{\beta }}_3^{\mathsf{T}}{U}_{3,r_1}^{\mathsf{T}}{r}_2\end{array}$$

$$\displaystyle\begin{split}\max\bigg{(}r_{1}^{\mathsf{T}}\big{(}\left(1-\bar{\alpha}_{1}\right)U_{1,r_{3}}+\left(1-\bar{\alpha}_{2}\right)U_{2,r_{3}}\\ +\left(1-\bar{\alpha}_{3}\right)U_{3,r_{3}}\big{)}r_{2}\\ -\bar{\beta}_{1}^{\mathsf{T}}U_{1,r_{3}}r_{2}-\bar{\beta}_{2}^{\mathsf{T}}U_{2,r_{3}}^{\mathsf{T}}r_{1}-\bar{\beta}_{3}^{\mathsf{T}}U_{3,r_{1}}^{\mathsf{T}}r_{2}\bigg{)}\end{split}$$

$$F_i{r}_i=f_i\forall i\in N$$

$$z\in \{0,1{\}}^{|L|}$$

$$\sum _{\ell \in L}z(\ell )=1$$

$$z_i\in \{0,1{\}}^{|L|}\forall i\in N$$

$$R_{i,(q_i,\cdot )}{z}_i=1\forall i\in N,\forall q_i\in {\bar{Q}}_i$$

$$z_i(\ell )\ge z(\ell )\forall i\in N,\forall \ell \in L$$

$$r_j\ge R_i{z}_i\forall i\in N,\forall j\in N\setminus \{i\}$$

$$r_i\ge 0\forall i\in N$$

Full text

Introduction

The computational study of adversarial interactions is a central problem in Artificial Intelligence, aiming at finding players’ optimal strategies and predicting the most likely outcome of a game. A vast body of literature focuses on the computation of Nash Equilibria (NEs), mainly in two-player zero-sum games (?). This setting is well understood and, recently, some remarkable results have been achieved by, e.g., ? (?; ?). While relevant, this model is rather restrictive, as many practical scenarios are not zero-sum and involve more than two players, and it presents some weaknesses when used as a prescriptive tool, in particular in general-sum games. Indeed, when multiple NEs coexist, the model assumes the lack of communication between the players, preventing them from synchronizing their strategies.

In practical situations where some form of communication is possible, solution concepts different from that of NE are required. The main alternative is the Correlated Equilibrium (CE), introduced by ? (?). In a CE, a device (i.e., a trusted external mediator) draws strategy profiles from a known joint probability distribution and privately communicates them to each player. The probability distribution induces an equilibrium if each player has no incentive to choose a different strategy from the recommended one, assuming the other players would not deviate either. A variation on the CE is the Coarse Correlated Equilibrium (CCE), introduced in (?), which only prevents deviations happening before knowing the device’s recommendation. In normal-form games, CEs and CCEs enjoy some appealing properties that make them plausible solution concepts in many practical scenarios. Specifically, they arises from simple and natural learning dynamics (?; ?), and they can be computed via linear programming on any normal-form game in polynomial time (assuming the number of players is fixed). Moreover, price-of-anarchy analyses show that coarse correlated equilibria characterizing outcomes of no-regret learning dynamics have near-optimal welfare (?; ?). While a CE can be found in polynomial time in some classes of succinctly representable multi-player games, finding an optimal CE in these games is, in general, NP-hard (?; ?). A similar result also holds for the problem of finding an optimal CCE. ? (?) show that for graphical, polymatrix, congestion, and anonymous games the problem is NP-hard.

Sequential games allow for richer forms of interaction among the players than normal-form games, which lead to different forms of correlation whose general understanding is still limited. Most of the works in this area focus on specific classes of games, such as Bayesian games (?; ?) and multi-stage games (?; ?). In these specific settings, the main solution concepts studied in the literature are the Normal-Form Correlated Equilibrium (NFCE), the Agent-Form Correlated Equilibrium (AFCE), and the Communication-Equilibrium. The first two equilibria only allow for a unidirectional communication from the device to the players, while the third equilibrium allows for bidirectional communication. The only known results for general extensive-form games are due to ? (?), who propose the notion of Extensive-Form Correlated Equilibrium (EFCE). The complex structure of extensive-form games significantly increases the computational effort required for correlation, as finding an optimal NFCE is $\mathsf{NP}$-hard even with two players (?). An optimal EFCE can be found efficiently in two-player games without Chance moves but, in games with three or more players (including Chance), finding an EFCE (or an AFCE) is $\mathsf{NP}$-hard (?). The only positive result for multi-player games is a polynomial-time algorithm to find an EFCE (?).

Correlated equilibria in which recommendations are drawn before the game starts are known as ex ante CEs. These equilibria require only unilateral communication from the device to the players. NFCE, AFCE, and EFCE belong to this family and differ in the time at which the recommendations are communicated to players. Specifically, the NFCE requires, for each player, a single interaction with the mediator taking place before the beginning of the game, whereas AFCE and EFCE require a message for each information set reached during the game. As a consequence, AFCE and EFCE are not suited for problems where the agents have limited communication capabilities, a situation which is frequent in practice. This is the case, for instance, of collusion in bidding, where communication during the auction is illegal, and coordinated swindling in public (see also the recent work by ? (?)). Different forms of correlation have been explored when a team of players faces an adversary (?; ?; ?; ?; ?). This setting, also known as ex ante coordination, is quite different from ours. Our notion of correlation is more flexible as any player may have different objectives. Therefore, in our correlation setting, individual players have to be incentivized to follow the recommendations of the mediator. In contrast, in the ex ante coordination setting there is no need for incentive constraints since team members share their final rewards.

Original Contributions

In this paper, we focus on equilibria requiring a low level of communication. A natural question is whether correlation can be reached efficiently when agents have limited communication capabilities, i.e., when they cannot receive messages during the execution of the game.111This rules out the possibility of employing an EFCE. Motivated by the hardness result for the NFCE, we introduce the notion of Normal-Form Coarse Correlated Equilibrium (NFCCE) as the extension of CCE to sequential games.

We prove that, unlike the NFCE, the problem of finding an optimal NFCCE admits a polynomial-time algorithm for two-player games without Chance moves. In particular, we devise a hybrid formulation (combining the normal and the sequence forms) for the problem of computing an optimal NFCCE featuring a polynomial number of constraints and an exponential number of variables. We then provide a polynomial-time separation oracle which, thanks to the ellipsoid algorithm (?), allows us to show that an optimal NFCCE can be computed in polynomial time. We also show that this approach cannot be extended to more general settings, illustrating that with more than two players, including Chance, the problem becomes NP-hard.

We describe a practical algorithm to compute an optimal NFCCE based on column generation—a variation of the simplex method in which the variables (columns) of the problem are introduced one at a time. We devise different oracles to solve the corresponding pricing problem. In particular, we provide a polynomial-time oracle suitable for the two-player setting, and an oracle based on a Mixed Integer Linear Program (MILP). Then, we show how to adapt the MILP oracle to the case of two-player games with Nature, and to general multi-player games.

Preliminaries

We briefly introduce several of the basic concepts we use in the rest of the paper. Further details can be found in (?).

Game Representations

An extensive-form game $\Gamma$ has a finite set of players $N$ and a finite set of actions $A$. Exogenous stochasticity is represented through a non-strategic player $c$ (the nature or chance player). $V$ is the set of non-terminal decision nodes, and $V_{i}\subseteq V$ is the set of decision nodes belonging to player $i\in N\cup\{c\}$. The set of terminal nodes (leaves) is denoted by $L$. The function $\iota:V\to N\cup\{c\}$ associates each decision node with the player acting at it. The function $\rho:V\to 2^{A}$ is the action function, assigning with each decision node a set of available actions. The successor function is denoted by $\chi:V\times A\to V\cup L$. Let $U_{i}:L\to\mathbb{R}$ be the utility function of each each $i\in N$. Moreover, let $U=\{U_{i}\}_{i\in N}$. Finally, for each $i\in N\cup\{c\}$ let $H_{i}$ be an information partition of $V_{i}$ such that decision nodes within the same information set $h\in H_{i}$ are not distinguishable by player $i$. We write $H=\{H_{i}\}_{i\in N\cup\{c\}}$. The a function $\pi_{c}$ is such that $\pi_{c}(h,a)$ is the fixed probability with which chance selects $a$ at $h\in H_{c}$. Moreover, $\rho(h)$ denotes the set of actions available at $h\in H_{i}$. We remark that, by definition, $\rho(x_{1})=\rho(x_{2})=\rho(h)$ for any player $i\in N\cup\{c\}$, information set $h\in H_{i}$, and $x_{1},x_{2}\in h$. In this paper, we focus on games with perfect recall, i.e., games where, at each stage, all the players recall all the information acquired at earlier stages.

An extensive-form game can be equivalently represented in normal-form. Let $P_{i}=\times_{h\in H_{i}}\rho(h)$ be the set of pure normal-form plans of player $i\in N$. A normal-form plan $p\in P_{i}$ specifies an action per information set of player $i$. The normal-form of an extensive-form game is characterized by the same set of players $N$, actions $P=\times_{i\in N}P_{i}$, and the set of utility functions $U^{\prime}=\{U_{i}^{\prime}\}_{i\in N}$. Function $U_{i}^{\prime}:P\to\mathbb{R}$ denotes the expected payoff obtained by marginalizing with respect to $\pi_{c}$. The reduced normal form is obtained by deleting duplicated strategies from the normal form.

Strategy Representations. A normal-form strategy $\sigma_{i}$ for $i\in N$ is defined as the function $\sigma_{i}:P_{i}\to\Delta^{|P_{i}|}$. We denote by $\Sigma_{i}$ the normal-form strategy space of player $i$. A correlated (joint) normal-form strategy $\sigma\in\Sigma$ is defined as $\sigma:P\to\Delta^{|P|}$. The size of a normal-form strategy is exponential in the size of the extensive-form tree. This shortcoming can be overcome by exploiting the sequence form (?), whose size is linear in the size of the game tree.

The sequence form decomposes strategies into sequences of actions and their realization probabilities. A sequence for player $i$, associated with a node $x$ of the game, is the subset of $A$ specifying player $i$’s actions on the path from the root to $x$. We denote the set of sequences of player $i$ by $Q_{i}$. A sequence is said terminal if it leads to a terminal node for at least a set of sequences of the other players. The set of terminal sequences of player $i$ is denoted by $\overline{Q}_{i}$. Moreover, we denote by $q_{\emptyset}$ the fictitious sequence leading to the root node and, for each action $a\in A$ and sequence $q\in Q_{i}$, we denote by $qa\in Q_{i}$ the extended sequence obtained by appending action $a$ to $q$.

A sequence-form strategy, said realization plan, is a function $r_{i}:Q_{i}\to\mathbb{R}$ associating each sequence $q\in Q_{i}$ with its probability of being played. A well-defined sequence-form strategy is such that $r_{i}(q_{\emptyset})=1$ for each $i\in N$ and, for each $h$ and sequence $q$ leading to $h$, $-r_{i}(q)+\sum_{a\in\rho(h)}r_{i}(qa)=0$ and $r_{i}(q)\geq 0$. These constraints are linear in the number of sequences and can be compactly written as $F_{i}\,r_{i}=f_{i}$, where $F_{i}$ is an $|H_{i}|\times|Q_{i}|$ matrix and $f_{i}^{\mathsf{T}}=(1,0,\ldots,0)$ is a vector of dimension $|H_{i}|$. The utility function of player $i$ is represented by a sparse $n$-dimensional matrix defined only for profiles of terminal sequences leading to a leaf node. With a slight abuse of notation, we denote it by $U_{i}\in\mathbb{R}^{|Q_{1}|\times\dots\times|Q_{n}|}$.

Correlation in Normal-Form Games

Let $p_{-i}=(p_{1},\ldots,p_{i-1},$ $p_{i+i},\ldots,p_{n})\in\times_{j\in N\setminus\{i\}}P_{j}$. The classical notion of CE (?) for normal-form games is:

Definition 1

$\sigma^{\ast}\in\Sigma$* is a correlated equilibrium of the normal form game $(N,P,U^{\prime})$ if, for every $i\in N$ and $p_{i},p_{i}^{\prime}\in P_{i}$, the following holds:*

[TABLE]

A CE can be interpreted in terms of a mediator who, ex ante the play, draws $(p_{1},\ldots,p_{n})$ according to the publicly known $\sigma^{\ast}$ and privately communicates each recommendation $p_{i}$ to the corresponding player.

Another possibility is enforcing protection against deviations of players which are independent from the sampled outcome. This can be done though the notion of coarse correlated equilibrium (?).

Definition 2

$\sigma^{\ast}\in\Sigma$* is a coarse correlated equilibrium of a normal-form game $(N,P,U^{\prime})$ if, for every $i\in N$ and $p_{i}^{\prime}\in P_{i}$, the following holds:*

[TABLE]

CCEs differ from CEs in that a CCE only requires that following the suggested action is a best response in expectation before the recommended action is actually revealed. Moreover, we recall that every CE is also a CCE while the converse is, in general, not true.

An optimal CCE may lead to a social welfare arbitrarily larger than the social welfare provided by the optimal CE on the same game. Figure 1 reports a normal-form game where this happens ($k>1$).

The joint strategy profile assigning probability $1/2$ to $(a_{1}^{1},a_{2}^{1})$ and $(a_{1}^{2},a_{2}^{2})$ is the CCE maximizing the social welfare of the players, which is $(k+1)/2$. The unique optimal CE is the probability distribution assigning probability 1 to $(a_{1}^{2},a_{2}^{2})$, providing a social welfare of 1 independently of $k$. Therefore, for increasing values of $k$, an optimal CCE allows the players to reach a social welfare which is arbitrarily larger than the social welfare reached through the optimal CE.

Correlation in General Extensive-Form Games

We review the main notions of correlation for general extensive-form games. In this general setting, it is customary to consider ex ante CEs, i.e., correlated equilibria in which an action profile is sampled before the game is played. In this paper, we focus on the following solution concepts:

Definition 3

A normal-form correlated equilibrium (normal-form coarse correlated equilibrium) of an extensive-form game $\Gamma$ is a correlated equilibrium (coarse correlated equilibrium) of the reduced normal-form game equivalent to $\Gamma$.

In these two solution concepts, the entire vector of recommendations specifying one action per information set is revealed to the players before the game starts. Thus, once the recommendation is received each player commits to playing a pure strategy.

Informally, an AFCE (?) is a CE of the agent-form game equivalent to the given extensive-form game. In the agent form of the game, moves are chosen by a different agent per information set of the player. In an EFCE (?), each recommendation is assumed to be in a sealed envelope and is revealed only when the player reaches the relevant information set (i.e., the information set where she can make that move). The main difference between EFCE and NFCE/NFCCE is that the former requires recommendations to be delivered during the game execution, thus being more demanding in terms of communication requirements. It is crucial to notice that the size of the signal that has to be sampled is the same, and it has polynomial size (one action per information set).

Letting $S_{\circ}$ be the set of equilibria of type $\circ$ of a given game, we have: $S_{NFCE}\subseteq S_{EFCE}\subseteq S_{NFCCE}\subseteq S_{AFCE}$. See ? (?) for further details.

In the next section, we study the problems of computing an NFCE and an NFCCE maximizing the social welfare (i.e., the cumulative utility of the players). We refer to them as NFCE-SW and NFCCE-SW. The generalization of our results to the case in which one searches for an equilibrium maximizing a linear combination of the players’ utility, omitted here for reasons of space, is straightforward.

Complexity of an Optimal NFCCE

We show that there exists a polynomial-time algorithm for solving the NFCCE-SW problem with two players. First, we provide a compact formulation for the problem. Then, we describe a polynomial-time algorithm for solving it.

Problem Formulation

Given an extensive-form game $\Gamma$, a direct application of Definition 3 yields a Linear Programming problem (LP) with an exponential number of variables and an exponential number of constraints. We provide the following result:

Lemma 1

The NFCCE-SW problem for an extensive-form game $\Gamma$ can be formulated as an LP with an exponential number of variables but only a polynomial number of constraints.

To prove the lemma, we provide a hybrid representation which exploits the tree structure of the problem combining both the normal form and the sequence form. Let $r_{p_{i}}\in\{0,1\}^{|Q_{i}|}$ be a $|Q_{i}|$-dimensional column vector representing the pure realization plan for player $i\in N$ that is realization equivalent to $p_{i}\in P_{i}$.222A realization plan is realization equivalent to a normal-form plan if, for any strategy profile of the other players, they enforce the same probability distribution over the terminal nodes of the game tree. We recall that every plan of the reduced normal form is realization equivalent to exactly one pure realization plan, see ? (?). In the following and when not differently specified, $U_{i}$ denotes the sequence-form utility matrix of player $i$.

According to Definition 2, the constraints describing an NFCCE for Player $1$ can be written as follows (for Player $2$, the constraints are analogous):

[TABLE]

The first term is the expected utility of Player $1$ at the equilibrium. Let $v_{1}$ be the $|H_{1}|$-dimensional vector of variables of the dual of the best-response problem in sequence form. By definition of sequence form, $f_{1}^{\mathsf{T}}\,v_{1}$ is equal to the first component of $v_{1}$, whose value corresponds to the utility of Player $1$ at the equilibrium. Then:

[TABLE]

The second term of the above inequalities can be written as

[TABLE]

Letting $\bar{\sigma}_{2}(p_{2})=\sum_{p_{1}\in P_{1}}\sigma(p_{1},p_{2})$, $\bar{\sigma}_{2}\in\Delta^{|P_{2}|}$ can be interpreted as the prior probability with which plan $p_{2}$ is played by Player $2$. $\bar{\sigma}_{2}$ can be written as the following realization-equivalent sequence-form strategy: $\bar{r}_{2}=\sum_{p_{2}\in P_{2}}\bar{\sigma}(p_{2})r_{p_{2}}$, which is a valid realization plan due to convexity. Now, we only need to show that $f_{1}^{\mathsf{T}}\,v_{1}$ is not strictly smaller than the value of the best response of Player $1$ given the strategy $\bar{r}_{2}$ of Player $2$. By exploiting the dual of the best-response problem in sequence form, this is equivalent to showing $F_{1}^{\mathsf{T}}v_{1}-U_{1}\,\bar{r}_{2}\geq 0$. Thus, expanding $\bar{r}_{2}$ and deriving the equilibrium constraints for Player $2$ we obtain the following mathematical program:

[TABLE]

This formulation constitutes a proof of Lemma 1 as it employs a polynomial number of constraints (namely, $|Q_{1}|+|Q_{2}|+3$) and an exponential number of variables.

Efficient Algorithm

The following lemma will be employed to prove our central result. It shows that a player can reason in a best-response fashion to minimize the utility of the other player weighted by an arbitrary distribution, while also guaranteeing the reachability of a given terminal node.

Lemma 2

Given a generic two-player extensive-form game $\Gamma$, an outcome $\ell\in L$, and a vector $\zeta\in\mathbb{R}^{|Q_{1}|}$, the problem of finding $p_{2}\in P_{2}$ under the constraints that

• •

there exists some $p_{1}\in P_{1}$ s.t. $(p_{1},p_{2})$ leads to outcome $\ell$ and

• •

$\zeta^{\mathsf{T}}\,U_{1}\,r_{p_{2}}$* is minimized*

can be solved in polynomial time. The same holds when the two players are interchanged.

Proof. Let us focus on the case in which we look for $p_{2}\in P_{2}$. First, define $\bar{U}_{1}$ s.t. $\bar{U}_{1}(q_{1},q_{2}):=\zeta(q_{1})U_{1}(q_{1},q_{2})$ for each $(q_{1},q_{2})\in Q_{1}\times Q_{2}$. Then, let $\bar{\Gamma}$ be the extensive-form game obtained from $\Gamma$ by substituting Player 1’s utility function with $\bar{U}_{1}$. Given $\bar{\Gamma}$, denote by $(q_{1}^{\ell},q_{2}^{\ell})$ the pair of sequences identifying $\ell$, and by $H_{i}^{\ell}$ the set of information sets of player $i$ encountered in sequence $q_{i}^{\ell}$. Algorithm 1 returns the set of actions ($A^{\prime}_{i}$) forming a plan of the normal-form game (not reduced) equivalent to $\bar{\Gamma}$.

To retrieve $A^{\prime}_{i}$, Algorithm 1 performs a depth-first traversal of the tree while keeping track of the value to be minimized at each decision node ($\nu$) and selecting actions while moving backwards. Then, $p_{2}$ can be computed by traversing the tree from the root, and selecting actions according to those specified in $A^{\prime}_{2}$. $\Box$

Let us focus on the dual $\mathcal{D}$ of LP (1)–(6):

Lemma 3

$\mathcal{D}$* admits a polynomial-time separation oracle.*

Proof. Let $\alpha_{i}\in\mathbb{R}$, for all $i\in N$, be the dual variables of constraints (2), $\beta_{1}\in\mathbb{R}^{|Q_{1}|}$ the dual variables of constraints (3), $\beta_{2}\in\mathbb{R}^{|Q_{2}|}$ the dual variables of constraints (4), and $\gamma\in\mathbb{R}$ the dual variable of constraint (5). With $n=2$, $\mathcal{D}$ is an LP with a number of variables ($|Q_{1}|+|Q_{2}|+3$) polynomial in the size of the tree and an exponential ($|P_{1}\times P_{2}|+|H_{1}|+|H_{2}|$) number of constraints. We show that, given a vector $\bar{z}=(\bar{\alpha}_{1},\bar{\alpha}_{2},\bar{\beta}_{1},\bar{\beta}_{2},\bar{\gamma})$, the problem of either finding a hyperplane separating $\bar{z}$ from the set of feasible solutions to $\mathcal{D}$ or proving that no such hyperplane exists can be solved in polynomial time. Since the number of dual constraints corresponding to the primal variables $v_{i}$ is linear, these constraints can be checked efficiently for violation. We are left with the problem of determining whether any of the following constraints, defined for all $(p_{1},p_{2})\in P_{1}\times P_{2}$, is violated:

[TABLE]

Let us consider the separation problem of finding an inequality of $\mathcal{D}$ which is maximally violated at $\bar{z}$. The problem reads:

[TABLE]

A pair $p_{1},p_{2}$ yielding a violated inequality exists iff the separation problem admits an optimal solution of value $<-\bar{\gamma}$.

One such pair (if any) can be found in polynomial time by enumerating over the (polynomially many) possible outcomes $\ell\in L$ of the game. For each of them, we look for the pair $(p_{1}^{\ell},p_{2}^{\ell})$ minimizing the objective function of the separation problem, halting as soon as a pair $(p_{1}^{\prime},p_{2}^{\prime})$ yielding a violated constraint is found. If the procedure terminates without finding any suitable pair, we deduce that no violated inequalities exist and $\mathcal{D}$ has been solved. First, notice that $r^{\mathsf{T}}_{p_{1}}((\bar{\alpha}_{1}-1)U_{1}+(\bar{\alpha}_{2}-1)U_{2})r_{p_{2}}$ is constant for the family of pairs identifying $\ell\in L$. Therefore, we can consider an individual subproblem for each player (i.e., we can find $p_{1}^{\ell}$ and $p_{2}^{\ell}$ independently). Hence, for each outcome $\ell$ and for each player $i$ the corresponding $p_{i}^{\ell}$ can be found in polynomial time due to Lemma 2. $\Box$

The following theorem shows that, in certain cases, the NFCCE-SW problem can be solved efficiently:

Theorem 4

Given an extensive-form game $\Gamma$ with $n=2$ players and without chance moves, an NFCCE maximizing the social welfare can be computed in time polynomial in the size of the game tree.

Proof. Lemma 3 shows that there exists a polynomial-time separation oracle for $\mathcal{D}$. Then, $\mathcal{D}$ can be solved in polynomial time via the ellipsoid method due to the equivalence between optimization and separation (?; ?). As the method solves, in polynomial time, a primal-dual system encompassing not just $\mathcal{D}$ but also its primal problem NFCEE-SW, it also produces, simultaneously, an optimal solution to the latter. $\Box$

Negative Result

The approach that we presented here cannot be extended to games with two players and the chance player as, upon introducing the latter, the problem transitions from polynomially solvabile to NP-hard:333Other problems in which this transition takes place are, for example, the problem of computing a socially optimal EFCE (?) and the problem of deciding if a two-player zero-sum extensive-form game with perfect recall admits a pure strategy equilibrium (?; ?).

Theorem 5

Computing an NFCCE maximizing the social welfare is NP-hard even in extensive-form games with two players, chance moves, and binary outcomes.

Proof Sketch. A construction introduced by ? (?) can be employed. The reduction is from SAT, whose generic instance is a Boolean formula $\phi$ in conjunctive normal form with $\eta$ clauses and $\nu$ variables. Given $\phi$, we build an auxiliary game $\Gamma_{\phi}$, of size proportional to that of the boolean formula, following (?, Theorem 1.3). $\Gamma_{\phi}$ admits a pure strategy guaranteeing a social welfare of 2 if and only if $\phi$ is satisfiable. Otherwise, the maximum expected social welfare cannot be more than $2(1-1/\eta)$. A pure strategy maximizing the social welfare is also an NFCCE, since no ex ante deviation would result in an increase in the player’s utility, being it already maximal. Then, finding a solution to NFCCE-SW in polynomial time would imply the existence of a polynomial time algorithm for SAT, which leads to a contradiction, unless P=NP. $\Box$

Notice that, when considering the separation problem of $\mathcal{D}$, working with chance is hard because the first term of the objective function of the separation problem is no longer constant when the outcome is fixed. In the case with $n>2$ and no chance moves, one would have to determine the joint best response of two player a time (to maximize the terms of the objective function of the separation problem following the first one), which is NP-hard (?).

A Practical Algorithm

Due to being based on the ellipsoid method (which, while being a powerful theoretical tool, is well-known to be inefficient in practice), the algorithm that we used in the proof of Theorem 4 is not appealing from a practical perspective. We propose, here, a computationally more efficient method based on the simplex method to compute optimal NFCCEs via a column generation technique. The focus on two-player games is motivated by the negative result in the previous section.

Let $x$ be a vector containing the variables of LP (1)–(6):

[TABLE]

where, for each $i=1,2$, $v_{i}$ is defined as in the proof of Lemma 1 and $s_{i}$ is a $|Q_{i}|$-dimensional column vector of slack variables. The cost vector $c$ associated with the variables is:

[TABLE]

where $U_{i}^{\prime}$ is the utility matrix of the reduced normal-form game. We compactly rewrite the constraints of LP (1)–(6) in standard form as $M\,x=b$, where $b^{\mathsf{T}}=(1,0,\ldots,0)$ is a vector of dimension $(|Q_{1}|+|Q_{2}|+3)$. We denote the $j$-th column of $M$ by $M_{(\cdot,j)}$.

The algorithm works in two phases, determining, first, a basic feasible solution and, then, iteratively improving it until an optimal one is found. The crucial component of the algorithm is an oracle for solving, given a basic feasible solution to LP (1)–(6), the problem (we refer to it as LRC) of finding a variable with the largest reduced cost. Notice that Theorem 4 already implies the tractability of the problem of finding the variable with the maximum reduced cost—the so-called (primal) pricing problem, as it is equivalent to finding a maximally violated constraint in the dual $\mathcal{D}$. Hence:

Corollary 5.1

LRC can be solved in polynomial-time.

Letting $c_{j}$ be the cost associated with the $j$-th component of $x$ and letting $c_{B}$ be the vector of costs of the basic variables, the $j$-th reduced cost is:

[TABLE]

where $B=[M_{(\cdot,j^{\prime})},M_{(\cdot,j^{\prime\prime})},\ldots]$ for each index $j^{\prime},j^{\prime\prime},\ldots$ corresponding to a basic variable. We rely on the following polynomial-time oracle, P-LRC, described in Algorithm 2 (another oracle is presented in the next section).

First, notice that, given a basic feasible solution, $c_{B}^{\mathsf{T}}B^{-1}$ is equal to a vector (call it $\zeta$) of dimension $(|Q_{1}|+|Q_{2}|+3)$, computable in polynomial time (Line 4). By employing the same notation as the one adopted for the dual variables in the proof of Lemma 3, $\zeta^{\mathsf{T}}=(\bar{\beta}_{1},\bar{\beta}_{2},\bar{\alpha}_{1},\bar{\alpha}_{2},\bar{\gamma})$, where $\bar{\beta}_{i}$ is the vector of dual variables of constraints (3) and (4), $\bar{\alpha}_{i}$ are the dual variables of constraints (2), and $\bar{\gamma}$ is that of constraint (5).

The reduced costs of the variables $v_{i}$ and $s_{i}$ can be computed directly by definition since their number is polynomial in the size of the tree (Lines 5 to 7). We are left with the problem of evaluating the reduced costs of the $\sigma(\cdot,\cdot)$ variables. P-LRC enumerates the outcomes of the game (Line 8). Since all the pairs of plans identifying $\ell$ have the same $c_{j}$, the problem of minimizing $\zeta^{\mathsf{T}}M_{(\cdot,j)}$ amounts to finding a pair $(p_{1},p_{2})$ minimizing $(\bar{\beta}_{1}U_{1}r_{p_{2}}+\bar{\beta}_{2}U_{2}^{\mathsf{T}}r_{p_{1}})$. The problem can be split into a subproblem per player, and solved through Algorithm 1, which we presented in the proof of Lemma 2 (Line 9, where we simplified the signature of C-PLAN-SEARCH for ease of notation). By applying this procedure for each of the outcomes and selecting, among the resulting pairs, the one with the largest reduced cost (Line 13), we are able to determine the new variable entering the basis in polynomial time.

The two phases of the overall algorithm are the following ones, and both adopt P-LRC:

Phase 1: finding a feasible point. A basic feasible solution to NFCCE-SW is determined through an auxiliary problem with artificial variables, where a new variable is introduced for each equality constraint, and their sum is minimized in the objective function. If some artificial variable with index $\bar{j}$ is found in the optimal basis of the auxiliary problem, we can find, in polynomial-time, a variable $j$ of the original problem to replace it by either maximizing or minimizing $e_{\bar{j}}B^{-1}M_{(\cdot,j)}$, where $e_{\bar{j}}$ is a vector of zeros with suitable dimension and equal to 1 in position $j$ (the problem can be solved with Algorithm 1).

Phase 2: finding an optimal solution. Starting from a basic feasible solution, the algorithm iteratively improves it until an optimal solution is found. While, if we were to solve the problem with a standard implementation of the simplex method, we would have to compute the reduced cost of all the nonbasic variables to find one to enter the basis (which would require exponential time in the size of the game), by employing P-LRC the next variable to enter the basis can be found in polynomial time. This follows from the same reasoning that led to Corollary 5.1.

We remark that, while the two phases require polynomial time, the bottleneck of the approach is that, at each iteration, P-LRC has to traverse the game tree twice for each $\ell\in L$. To circumvent this issue, we present a second oracle based on mixed-integer linear programming (see the experimental evaluation for a comparison between the two approaches).

General Mixed-Integer Oracle

In this section, we describe an oracle (MI-LRC) for computing a solution to LRC by solving a Mixed-Integer Linear Program (MILP). Differently from P-LRC, MI-LRC does not need the explicit enumeration of the terminal nodes of the game, and, furthermore, it can be extended to games with chance and more that two players. We provide, here, a description of the oracle for the case of a two-player game with and without chance moves.444MI-LRC can be extended to games with $n>2$, we omit the description of this setting due to space constraints.

The crucial difference between MI-LRC and P-LRC is in the way they handle the inspection of the reduced costs associated with the $\sigma(\cdot,\cdot)$ variables. In MI-LRC, lines 8–12 of Algorithm 2 are substituted with an MILP.

Two-player games

Let us first focus on the case of a two-player game without chance moves. Let $R_{i}$ be a $|Q_{i}|\times|L|$ matrix such that $R_{i}(q_{i},\ell)=1$ if $q_{i}$ is on the path from the root to $\ell$, and $R_{i}(q_{i},\ell)=0$ otherwise. Let also $z$ be an $|L|$-dimensional vector of binary variables. MI-LRC solves the following problem:

[TABLE]

The objective function (7) follows from the definition of the reduced costs (we are looking for a variable whose dual constraint is maximally violated). Constraints (9) force the realization plans to select with probability 1 the sequences on the path to the selected outcome $\ell$. Notice that, while the objective function contains quadratic terms, they only involve binary variables. Therefore, it can be restated as a linear function after introducing a new variable and four linear constraints per bilinear term according to the formulation proposed in (?).

Notice that an optimal realization plan $r^{\ast}_{i}$, solution to MI-LRC, may not be pure (i.e., there may exist some $q\in Q_{i}$ s.t. $r^{\ast}_{i}(q)\in(0,1)$). Nevertheless, there always exists a pair of pure realization plans leading to the same terminal node and granting the same value $\bar{\beta_{1}^{\mathsf{T}}}U_{1}r^{\ast}_{2}+r^{\ast\mathsf{T}}_{1}U_{2}\bar{\beta_{2}}$. Once a pair of pure realization plans has been determined, the reduced cost associated with it has to be computed according to equation (6) and compared to the reduced costs of the remaining variables (Line 13 of Algorithm 2).555It is enough to traverse the tree depth-first, and select sequences, among those played with strictly positive probability in $r_{i}^{\ast}$, following the same reasoning of Algorithm 1.

Two-player games with Nature

We denote by $(q_{1}^{\ell},q_{2}^{\ell},q_{c}^{\ell})$ the unique tuple of the sequences leading to $\ell$, where $q_{c}^{\ell}$ is a sequence of the chance player. The crucial point is that, given $\ell\in L$, there may exist some $\ell^{\prime}\in L\setminus\{\ell\}$, reachable through $(q_{1}^{\ell},q_{2}^{\ell},q_{c}^{\ell^{\prime}})$, satisfying $q_{c}^{\ell^{\prime}}\neq q_{c}^{\ell}$. MI-LRC can be adapted to this scenario as follows. First, for each $i\in N$ we compute the utility matrices $U_{i,\pi_{c}}$ (with dimension $|Q_{1}|\times|Q_{2}|$) obtained by marginalizing each $U_{i}$ with respect to $\pi_{c}$. Formally, denoting by $r_{c}$ the realization plan defined over the sequences of the chance player which are realization-equivalent to $\pi_{c}$, for each $(q_{1},q_{2})\in Q_{1}\times Q_{2}$ we have $U_{i,\pi_{c}}(q_{1},q_{2})=\sum_{q_{c}\in Q_{c}}r_{c}(q_{c})U_{i}(q_{1},q_{2},q_{c})$. Objective function (7) is then modified by substituting each $U_{i}$ with $U_{i,\pi_{c}}$. Moreover, upon denoting by $R_{c}$ the $|\bar{Q}_{c}|\times|L|$ matrix defined analogously to $R_{i}$, it suffices to substitute each of constraints (10), one per $\bar{q}\in\bar{Q}_{c}$, with the constraint $R_{c,(q_{c},\cdot)}z=1$, where $R_{c,(q_{c},\cdot)}$ denotes row $q_{c}$ of $R_{c}$. This way, MI-LRC can be extended to the more demanding setting of games with two-players and chance moves.

Multi-player games

We focus on a game with $n=3$, without chance moves. The oracle can be easily adapted to the setting with $n>3$, and to include the Chance player. Denote by $U_{i,r_{j}}$ the utility matrix of player $i$ marginalized with respect to realization plan $r_{j}$ of player $j$— notice that the $U_{i,r_{j}}\in\mathbb{R}^{|Q_{i}|\times|Q_{t}|}$, with $i,t\in N$, and $i,t\neq j$. The objective function that needs to be maximized is:

[TABLE]

The first term of the objective function only depends on the choice of a single terminal node $\ell\in L$. The following terms can be addressed following the same reasoning we employed to adapt MI-LRC to the case of a two-player game with Chance. For example, in $-\bar{\beta}_{1}^{\mathsf{T}}U_{1,r_{3}}r_{2}$ player 2 and 3 are jointly best-responding against a fixed distribution of player 1 ($\bar{\beta}_{1}$). Then, MI-LRC can be substituted with the following oracle:

[TABLE]

The oracle employs $n+1$ $|L|$-dimensional vectors of binary variables. Vector $z$ selects a single terminal node (constraint 15), determining the value of the first term of the objective function. Each $z_{i}$, instead, selects the terminal nodes reachable through a certain choice of plans of the players that are best-responding against $i$ (constraint 17). As an example, $z_{1}(\ell)=1$ iff $\ell$ is reachable through the chosen $(r_{2},r_{3})$. Realization plans are constrained to be consistent with the selected outcomes (constraint 19). Finally, the choices in $z$ and in each $z_{i}$ have to be mutually consistent (constraint 18).

Discussion

In this paper, we have studied ex ante correlated equilibria in extensive-form games with low communication requirements. First, we showed that an optimal NFCCE can be computed in polynomial time in two-player games. Moreover, we have devised a column generation method which allows for computing solutions iteratively, by employing one of the two oracles which we have devised for the problem of finding a column with the largest reduced cost. In the future, it would be interesting experimentally evaluate our techniques, and to eventually further improve the scalability of our methods to tackle practical problems. Among the possible techniques to achieve this, we mention the adoption of heuristics for solving our oracle, the use of stabilization techniques, and the introduction of dominance relationships among the columns.