On Monte-Carlo tree search for deterministic games with alternate moves and complete information

TL;DR

This paper introduces an optimal step-by-step Monte-Carlo Tree Search algorithm for deterministic, perfect-information games, aiming to improve minimax value estimation through adaptive match simulation, with theoretical and empirical analysis.

Contribution

It develops a novel, stepwise optimal MCTS algorithm that adaptively chooses simulations to maximize information gain about the minimax value, and analyzes its convergence and performance.

Findings

Algorithm converges in finite steps regardless of prior

Numerical results show limited overall improvement over standard MCTS

Some scenarios indicate potential efficiency of the proposed method

Abstract

We consider a deterministic game with alternate moves and complete information, of which the issue is always the victory of one of the two opponents. We assume that this game is the realization of a random model enjoying some independence properties. We consider algorithms in the spirit of Monte-Carlo Tree Search, to estimate at best the minimax value of a given position: it consists in simulating, successively, $n$ well-chosen matches, starting from this position. We build an algorithm, which is optimal, step by step, in some sense: once the $n$ first matches are simulated, the algorithm decides from the statistics furnished by the $n$ first matches (and the a priori we have on the game) how to simulate the $(n+1)$-th match in such a way that the increase of information concerning the minimax value of the position under study is maximal. This algorithm is remarkably quick. We prove that our step by step optimal algorithm is not globally optimal and that it always converges in a finite number of steps, even if the a priori we have on the game is completely irrelevant. We finally test our algorithm, against MCTS, on Pearl's game and, with a very simple and universal a priori, on the games Connect Four and some variants. The numerical results are rather disappointing. We however exhibit some situations in which our algorithm seems efficient.

Figures

Equations

$$R(x)=1_{\{t(x)=0\}}\min \{R(y):y\in {\mathcal{C}}_x\}+1_{\{t(x)=1\}}\max \{R(y):y\in {\mathcal{C}}_x\}.$$

$$(({\mathcal{T}}_x,(R(y){)}_{y\in \mathcal{L}\cap {\mathcal{T}}_x}),x\in L_T)$$

$$s(T,z)=\mathbb{E}\Big{[}\Big{(}\Pr(R(z)=1|{\mathcal{G}}\lor\sigma(A_{T}))-m(T,z)\Big{)}^{2}\Big{|}A_{T}\Big{]}.$$

$$R_{n}(x)={{\mathbf{1}}}_{\{t(x)=0\}}\prod_{y\in{\mathcal{C}}_{x}}R_{n}(y)+{{\mathbf{1}}}_{\{t(x)=1\}}\Big{(}1-\prod_{y\in{\mathcal{C}}_{x}}(1-R_{n}(y))\Big{)}.$$

$$U_n(x)=1_{\{t(f(x))=0\}}\prod _{y\in {\mathcal{H}}_x}{R}_n(y)+1_{\{t(f(x))=1\}}\prod _{y\in {\mathcal{H}}_x}(1-R_n(y)).$$

$$Z_n(x)=\max \{(U_n(y){)}^2{Z}_n(y):y\in {\mathcal{C}}_x\}.$$

$$z_{*}=\arg\!\min\Big{\{}\mathbb{E}\Big{[}(R_{n+1}^{z}(r)-R(r))^{2}\Big{|}{\mathcal{F}}_{n}\Big{]}:z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\Big{\}}.$$

$$z_{*}=\arg\!\min\Big{\{}\mathbb{E}\Big{[}|R_{n+1}^{z}(r)-R(r)|\Big{|}{\mathcal{F}}_{n}\Big{]}:z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\Big{\}}.$$

$$m({\mathcal{K}}_x,x)=$$

$$s({\mathcal{K}}_x,x)=$$

$$\mathbb{E}[({\tilde{R}}_2(r)-R(r){)}^2]>\mathbb{E}[(R_2(r)-R(r){)}^2]but\mathbb{E}[({\tilde{R}}_3(r)-R(r){)}^2]<\mathbb{E}[(R_3(r)-R(r){)}^2].$$

$$R_n(x)\in \{0,1\}$$

$$z=r$$

$$m(S,x)=$$

$$s(S,x)=$$

$$\Theta (S,$$

$$\Gamma (S,$$

$$\displaystyle+{{\mathbf{1}}}_{\{t(x)=1\}}\Big{(}s(S\cup{{\mathbf{y}}},y)\!\!\prod_{u\in{{\mathbf{y}}}\setminus\{y\}}[1-m(S\cup{{\mathbf{y}}},u)]^{2}+\Big{[}\!\!\prod_{u\in{{\mathbf{y}}}}(1-m(S\cup{{\mathbf{y}}},u))-(1-m(S,x))\Big{]}^{2}\Big{)}.$$

$$m(S,x)=\mathrm{Pr}(x\in \mathcal{L},R(x)=1|A_S)+\sum _{\mathbf{y}\subset {\mathbb{C}}_x,|\mathbf{y}|\ge 1}\mathrm{Pr}({\mathcal{C}}_x=\mathbf{y},R(x)=1|A_S).$$

$$\mathrm{Pr}(R(x)=1|A_S\cap \{{\mathcal{C}}_x=\mathbf{y}\})=\Theta (S,x,\mathbf{y}).$$

$$\mathrm{Pr}(R(x)=1|A_S\cap \{{\mathcal{C}}_x=\mathbf{y}\})=1-\prod _{y\in \mathbf{y}}\mathrm{Pr}(R(y)=0|A_S\cap \{{\mathcal{C}}_x=\mathbf{y}\}).$$

$$s(S,x)=\mathbb{E}[(R(x)-m(S,x){)}^2{1}_{\{x\in \mathcal{L}\}}|A_S]+\sum _{\mathbf{y}\subset {\mathbb{C}}_x,|\mathbf{y}|\ge 1}\mathbb{E}[(R_1(x)-m(S,x){)}^2{1}_{\{{\mathcal{C}}_x=\mathbf{y}\}}|A_S].$$

$$\mathbb{E}[(R_1(x)-m(S,x){)}^2|A_S\cap \{{\mathcal{C}}_x=\mathbf{y}\}\cap \{w=y\}]=\Gamma (S,x,\mathbf{y},y).$$

$$R_{1}(x)=\Pr\Big{(}\prod_{u\in{{\mathbf{y}}}}R(u)=1\Big{|}{\mathcal{G}}\lor\sigma(A_{S\cup{{\mathbf{y}}}})\Big{)}=\Pr(R(y)=1|{\mathcal{G}}\lor\sigma(A_{S\cup{{\mathbf{y}}}}))\prod_{u\in{{\mathbf{y}}}\setminus\{y\}}\Pr(R(u)=1|A_{S\cup{{\mathbf{y}}}}).$$

$$R_{1}(x)-m(S,x)=[R_{1}(y)-m(S\cup{{\mathbf{y}}},y)]\prod_{u\in{{\mathbf{y}}}\setminus\{y\}}m(S\cup{{\mathbf{y}}},u)+\Big{[}\prod_{u\in{{\mathbf{y}}}}m(S\cup{{\mathbf{y}}},u)-m(S,x)\Big{]}.$$

$$\mathbb{E}[R_1(y)|A_S\cap \{{\mathcal{C}}_x=\mathbf{y}\}\cap \{w=y\}]=$$

$$=$$

$$\Gamma(U_{x},x,{{\mathbf{y}}},y)=s(U_{y},y)\!\!\prod_{u\in{{\mathbf{y}}}\setminus\{y\}}m(U_{u},u)+\Big{[}\!\!\prod_{u\in{{\mathbf{y}}}}m(U_{u},u)-m(U_{x},x)\Big{]}^{2}=0.$$

$$A_n=\{{\mathbf{x}}_n={\mathbf{y}}_n,{\mathcal{D}}_{{\mathbf{y}}_n}=D_n,(R(y){)}_{y\in {\mathbf{y}}_n}=(a(y){)}_{y\in {\mathbf{y}}_n}\}$$

$$A_n^{\prime }=\{{\mathbf{y}}_n\subset \mathcal{L},{\mathcal{D}}_{{\mathbf{y}}_n}=D_n,(R(y){)}_{y\in {\mathbf{y}}_n}=(a(y){)}_{y\in {\mathbf{y}}_n}\}$$

$$\frac{\mathrm{Pr}(\mathcal{T}=T,(R(y){)}_{y\in L_T}=(\alpha (y){)}_{y\in L_T},x_1=y_1)}{\mathrm{Pr}({\mathcal{D}}_{{\mathbf{y}}_1}=D_1,R(y_1)=a(y_1),x_1=y_1)}=\frac{\mathrm{Pr}(\mathcal{T}=T,(R(y){)}_{y\in L_T}=(\alpha (y){)}_{y\in L_T})}{\mathrm{Pr}(y_1\in \mathcal{L},{\mathcal{D}}_{{\mathbf{y}}_1}=D_1,R(y_1)=a(y_1))}$$

$$p=$$

Full text

We warmly thank Bruno Scherrer for his invaluable help. We also thank the anonymous referee for his numerous fruitful comments.

1. Introduction

1.1. Monte-Carlo Tree Search algorithms

Monte-Carlo Tree Search (MCTS) are popular algorithms for heuristic search in two-player games. Let us mention the book of Munos [19] and the survey paper of Browne et al. [7], which we tried to briefly summarize here and to which we refer for a much more complete introduction on the topic.

We consider a deterministic game with complete information and alternate moves involving two players, that we call $J_{1}$ and $J_{0}$. We think of Go, Hex, Connect Four, etc. Such a game can always be represented as a discrete tree, of which the nodes are the positions of the game. Indeed, even if a single position can be thought as the child of two different positions, we can always reduce to this case by including the whole history of the game in the position. Also, we assume that the only possible outcomes of the game, which are represented by values on the leaves of the tree, are either $1$ (if $J_{1}$ wins) or [math] (if $J_{0}$ wins). If draw is a possible outcome, we e.g. identify it to a victory of $J_{0}$.

Let $r$ be a configuration in which $J_{1}$ has to choose between several moves. The problem we deal with is: how to select at best one of these moves with a computer and a given amount of time.

If having a huge amount of time, such a question can classically be completely solved by computing recursively, starting from the leaves, the minimax values $(R(x))_{x\in{\mathcal{T}}}$, see Remark 1. Here ${\mathcal{T}}$ is the tree (with root $r$ and set of leaves ${\mathcal{L}}$) representing the game when starting from $r$, and for each $x\in{\mathcal{T}}$, $R(x)$ is the value of the game starting from $x$. In other words, $R(x)=1$ if $J_{1}$ has a winning strategy when starting from $x$ and [math] else. So we compute $R(x)$ for all the children $x$ of $r$ and choose a move leading to some $x$ such that $R(x)=1$, if such a child exists.

In practice, this is not feasible, except if the game (starting from $r$) is very small. One possibility is to cut the tree at some reasonable depth $K$, to assign some estimated values to all positions of depth $K$, and to compute the resulting (approximate) minimax values on the subtree above depth $K$. For example if playing Connect Four, one can assign to a position the value remaining number of possible alignments for $J_{1}$ minus remaining number of possible alignments for $J_{0}$. Of course, the choice of such a value is highly debatable, and heavily depends on the game.

A more universal possibility, introduced by Abramson [1], is to use some Monte-Carlo simulations: from each position with depth $K$, we handle a certain number $N$ of uniformly random matches (or matches with a simple default policy), and we evaluate this position by the number of these matches that led to a victory of $J_{1}$ divided by $N$. Such a procedure is now called Flat MCTS, see Coquelin and Munos [10], Browne et al. [7], see also Ginsberg [15] and Sheppard [21].

Coulom [11] introduced the class of MCTS algorithms. Here are the main ideas: we have a default policy and a selection procedure. We make $J_{1}$ play against $J_{0}$ a certain number of times and make grow a subtree of the game. Initially, the subtree ${\mathcal{T}}_{0}$ consists of the root and its children. After $n$ steps, we have the subtree ${\mathcal{T}}_{n}$ and some statistics $(C(x),W(x))_{x\in{\mathcal{T}}_{n}}$ provided by the previous matches: $C(x)$ is the number of times the node $x$ has been crossed and $W(x)$ is the number of times this has led to a victory of $J_{1}$. Then we select a leave $y$ of ${\mathcal{T}}_{n}$ using the selection procedure (which relies on the statistics $(C(x),W(x))_{x\in{\mathcal{T}}_{n}}$) and we end the match (from $y$) by using the default policy. We then build ${\mathcal{T}}_{n+1}$ by adding to ${\mathcal{T}}_{n}$ the children of $y$, and we increment the values of $(C(x),W(x))_{x\in{\mathcal{T}}_{n+1}}$ according to the issue of the match (actually, it suffices to compute these values for $x$ in the branch from $r$ to $y$ and for the children of $y$). Once the given amount of time is elapsed, we choose the move leading to the child $x$ of $r$ with the highest $W(x)/C(x)$.

Actually, this procedure throws away a lot of data: $C(x)$ is not exactly the number of times $x$ has been crossed, it is rather the number of times it has been crossed since $x\in{\mathcal{T}}_{n}$, and a similar fact holds for $W(x)$. In practice, this limits the memory used by the algorithm.

The most simple and universal default policy is to choose each move at uniform random and this is the case we will study in the present paper. It is of course more efficient to use a simplified strategy, depending on the game under study, but this is another topic.

Another important problem is to decide how to select the leave $y$ of ${\mathcal{T}}_{n}$. Kocsis and Szepesvári [17] proposed to use some bandit ideas, developed (and shown to be optimal, in a very weak sense, for bandit problems) by Auer et al. [3], see Bubeck and Cesa-Bianchi [5] for a survey paper. They introduced a version of MCTS called UCT (for UCB for trees, UCB meaning Upper Confidence Bounds), in which the selection procedure is as follows. We start from the root $r$ and we go down in ${\mathcal{T}}_{n}$: when in position $x$ where $J_{1}$ (resp. $J_{0}$) has to play, we choose the child $z$ of $x$ maximizing $W(z)/C(z)+\sqrt{c(\log n)/C(z)}$ (resp. $(C(z)-W(z))/C(z)+\sqrt{c(\log n)/C(z)}$). At some time we arrive at some leave $y$ of ${\mathcal{T}}_{n}$, this is the selected leave. Here $c>0$ is a constant to be chosen empirically. Kocsis and Szepesvári [17] proved the convergence of UCT.

Chaslot et al. [9] have broaden the framework of MCTS. Also, they proposed different ways to select the best child (after all the computations): either the one with the highest $W/C$, the one with the highest $C$, or something intermediate.

Gelly et al. [14] experimented MCTS (UCT) on Go. They built the program MoGo, which also uses some pruning procedures, and obtained impressive results on a $9\times 9$ board.

The early paper of Coquelin and Munos [10] contains many results. They showed that UCT can be inefficient on some particular trees and proposed a modification taking into account some possible smoothness of the tree and outcomes (in some sense). They also studied Flat MCTS.

Lee et al. [18] studied the problem of fitting precisely the parameters of the selection process. Of course, $W/C$ means nothing if $C=0$, so they empirically investigated what happens when using $(W+a)/(C+b)+\sqrt{c(\log n)/(C+1)}$, for some constants $a,b,c>0$. They conclude that $c=0$ is often the best choice. This is not so surprising, since the logarithmic term is here to prevent from large deviation events, which do asymptotically not exist for deterministic games. MoGo uses $c=0$ and ad hoc constants $a$ and $b$. This version of MCTS is the one presented in Section 9 and the one we used to test our algorithm.

Let us mention the more recent theoretical work by Buşoniu, Munos and Páll [6], as well as the paper of Garivier, Kaufmann and Koolen [13] who study in details a bandit model for a two-round two-player random game.

The survey paper of Browne et al. [7] discusses many tricks to improve the numerical results and, of course, all this has been adapted with very special and accurate procedures to particular games. As everybody knows, AlphaGo [22] became the first Go program to beat a human professional Go player on a full-sized board. Of course, AlphaGo is far from using only MCTS, it also relies on deep-learning and many other things.

1.2. Our goal

We would like to study MCTS when using a probabilistic model for the game. To simplify the problem as much as possible, we only consider the case where the default policy is play at uniform random, and we consider the modified version of MCTS described in Section 9, where we keep all the information. This may cause memory problems in practice, but we do not discuss such difficulties. So the modified version is as follows, for some constants $a,b>0$ to be fitted empirically. We make $J_{1}$ play against $J_{0}$ a certain number of times and make a subtree of the game grow. Initially, the subtree ${\mathcal{T}}_{0}$ consists in the root $r$ and its children. After $n$ steps, we have the subtree ${\mathcal{T}}_{n}$ and some statistics $(C(x),W(x))_{x\in{\mathcal{T}}_{n}}$ provided by the previous matches: $C(x)$ is the number of times the node $x$ has been crossed and $W(x)$ the number of times this has led to a victory of $J_{1}$. The $(n+1)$-th step is as follows: start from $r$ and go down in ${\mathcal{T}}_{n}$ by following the highest values of $(W+a)/(C+b)$ (resp. $(C-W+a)/(C+b)$) if it is $J_{1}$’s turn to play (resp. $J_{0}$’s turn to play), until we arrive at some leave $z$ of ${\mathcal{T}}_{n}$. From there, complete the match at uniform random until we reach a leave $y$ of ${\mathcal{T}}$. We then build ${\mathcal{T}}_{n+1}$ by adding to ${\mathcal{T}}_{n}$ the whole branch from $z$ to $y$ together with all the brothers of the elements of this branch, and we compute the values of $(C(x),W(x))_{x\in{\mathcal{T}}_{n+1}}$ (actually, it suffices to compute these values for $x$ in the branch from $r$ to $y$). Once the given amount of time is elapsed, we choose the move leading to the child $x$ of $r$ with the highest $C(x)/W(x)$.

As shown by Coquelin and Munos [10], one can build games for which MCTS is not very efficient. So it would be interesting to know for which class of games it is. This seems to be a very difficult problem. One possibility is to study if MCTS works well in mean, i.e. when the game is chosen at random. In other words, we assume that the tree and the outcomes are the realization of a random model. We use a simple toy model enjoying some independance properties, which is far from convincing if modeling true games but for which we can handle a complete theoretical study.

We consider a class of algorithms resembling the above mentioned modified version of MCTS. After $n$ simulated matches, we have some information ${\mathcal{F}}_{n}$ on the game: we have visited $n$ leaves, we know the outcomes of the game at these $n$ leaves, and we have the explored tree ${\mathcal{T}}_{n}=B_{n}\cup D_{n}$, where $B_{n}$ is the set of all crossed positions, and $D_{n}$ is the boundary of the explored tree (roughly, $D_{n}$ consists of uncrossed positions of which the father belongs to $B_{n}$).

So we can approximate $R(r)$, which is the quantity of interest, by $\mathbb{E}[R(r)|{\mathcal{F}}_{n}]$ (if the latter can be computed). Using only this information ${\mathcal{F}}_{n}$ (and possibly some a priori on the game furnished by the model), how to select $z\in D_{n}$ so that, simulating a uniformly random match starting from $z$ and updating the information, $\mathbb{E}[R(r)|{\mathcal{F}}_{n+1}]$ is as close as possible (in $L^{2}$) to $R(r)$?

We need a few assumptions. In words, (a) the tree and outcomes enjoy some independence properties, (b) we can compute, at least numerically, for $x\in D_{n}$, $m(x)=$ mean value of $R(x)$ and $s(x)=$ mean quantity of information that a uniformly random match starting from $x$ will provide. See Subsection 2.8 for precise definitions.

Under such conditions, we show that $\mathbb{E}[R(r)|{\mathcal{F}}_{n}]$ can indeed be computed (numerically), and $z$ can be selected as desired. The procedure resembles in spirit MCTS, but is of course more complicated and requires more computations.

This is extremely surprising: the computational cost to find the best $z\in D_{n}$ does not increase with $n$, because this choice requires to compute some values of which the update does not concern the whole tree ${\mathcal{T}}_{n}$, but only the last visited branch, as MCTS. (Actually, we also need to update all the brothers of the last visited branch, but this remains rather reasonable). It seems miraculous that this theoretical algorithm behaves so well. Any modification, such as taking draws into account or changing the notion of optimality, seems to lead to drastically more expensive algorithms, that require to update some values on the whole visited tree ${\mathcal{T}}_{n}$. We believe that this is the most interesting fact of the paper.

The resulting algorithm is explained in details in the next section. We will prove that this algorithm is convergent (in a finite number of steps) even if the model is completely irrelevant. This is not very surprising, since all the leaves of the game are visited after a finite number of steps. We will also prove on an example that our algorithm is myopic: it is not globally optimal. There is a theory showing that for a class of problems, a step by step optimal algorithm is almost globally optimal, i.e. up to some reasonable factor, see Golovin and Kraus [16]. However, it is unclear whether this class of problems includes ours.

1.3. Choice of the parameters

We will show, on different classes of models, how to compute the functions $m$ and $s$ required to implement our algorithm. We studied essentially two possibilities.

In the first one, we assume that the tree ${\mathcal{T}}$ is the realization of an inhomogeneous Galton-Watson tree, with known reproduction laws, and that the outcomes of the game are the realizations of i.i.d. Bernoulli random variables of which the parameters depend only on the depths of the involved (terminal) positions. Then $m(x)$ and $s(x)$ depend only on the depth of the node $x$. We can compute them numerically once for all, using rough statistics we have on the true game we want to play (e.g. Connect Four), by handling a high number of uniformly random matches.

The second possibility is much more universal and adaptive and works better in practice. At the beginning, we prescribe that $m(r)=a$, for some fixed $a\in(0,1)$. Then each time a new litter $\{y_{1},\dots,y_{d}\}$ (with father $x$) is created by the algorithm, we set $m(y_{1})=\dots=m(y_{d})=m(x)^{1/d}$ if $x$ is a position where it is $J_{0}$’s turn to play, and $m(y_{1})=\dots=m(y_{d})=1-(1-m(x))^{1/d}$ else. Observe here that $d$ is discovered by the algorithm at the same time as the new litter. Concerning $s$, we have different possibilities (see Subsection 2.14 and Section 6), more or less justified from a theoretical point of view, among which $s(x)=1$ for all $x$ does not seem to be too bad.

The first possibility seems more realistic but works less well than the second one in practice and requires some preliminary fitting. The second possibility relies on a symmetry modeling consideration: assuming that all the individuals of the new litter behave similarly necessarily leads to such a function $m$. This is much more universal in that once the value of $a=m(r)$ is fixed (actually, $a=0.5$ does not seem to be worse than another value), everything can be computed in a way not depending on the true game. Of course, the choice of $a=m(r)$ is debatable, but does not seem to be very important in practice.

1.4. Comments and a few more references

Our class of models generalize a lot the Pearl model [20], which simply consists of a regular tree with degree $d$, depth $K$, with i.i.d. Bernoulli$(p)$ outcomes on the leaves. However, we still assume a lot of independence. This is not fully realistic and is probably the main reason why our numerical experiments are rather disappointing.

The model proposed by Devroye-Kamoun [12] seems much more relevant, as they introduce correlations between the outcomes as follows. They consider that each edge of the tree has a value (e.g. Gaussian). The value of the leave is then $1$ if the sum of the values along the corresponding branch is positive, and [math] else. This more or less models that a player builds, little by little, his game. But from a theoretical point of view, it is very difficult to study, because we only observe the values of the leaves, not of the edges. So this unfortunately falls completely out of our scope. See also Coquelin and Munos [10] for a notion of smoothness of the tree, which seems rather relevant from a modeling point of view.

Finally, our approach is often called Bayesian, because we have an a priori law for the game. This has already been studied in the artificial intelligence literature. See Baum and Smith [4] and Tesauro, Rajan and Segal [24]. Both introduce some conditional expectations of the minimax values and formulas resembling (2) already appear.

1.5. Pruning

Our algorithm automatically proceeds to some pruning, as AlphaBeta, which is a clever version to compute exactly the minimax values of a (small) game. This can be understood if reading the proof of Proposition 15. The basic idea is as follows: if we know from the information we have that $R(x)=0$ for some internal node $x\in{\mathcal{T}}$ and if the father $v$ of $x$ is a position where $J_{0}$ plays, then there is no need to study the brothers of $x$, because $R(v)=0$. And indeed, our algorithm will never visit these brothers.

Some versions of MCTS with some additional pruning procedures have already been introduced empirically. See Gelly et al. [14] for MoGo, as well as many other references in [7]. It seems rather nice that our algorithm automatically prunes and this holds even if the a priori we have on the game is completely irrelevant. Of course, if playing a large game, this pruning will occur only near the leaves.

1.6. Numerical experiments

We have tested our general algorithm against some general versions of MCTS. We would not have the least chance if using some versions of MCTS modified in such a way that it takes into account some symmetries of a particular game, with a more clever default policy, etc. But we hope our algorithm might also be adapted to particular games.

Next, let us mention that our algorithm is subjected to some numerical problems, in that we have to compute many products, that often lead numerically to [math] or $1$. We overcome such problems by using some logarithms, which complicates and slows down the program.

Let us now briefly summarize the results of our experiments, see Section 8.

We empirically observed on various games that, very roughly, each iteration of our algorithm requires between $2$ and $4$ times more computational time than MCTS.

When playing Pearl’s games, our algorithm seems rather competitive against MCTS (with a given amount of time per move), which is not very surprising, since our algorithm is precisely designed for such games.

We also tried to play various versions of Connect Four. Globally, we are clearly beaten by MCTS. However, there are two situations where we win.

The first one is when the game is so large, or the amount of time so small, that very few iterations can be performed by the challengers. This is quite natural, because (a) our algorithm is only optimal step by step, (b) it relies on some independence properties that are less and less true when performing more and more iterations.

The second one is when the game is so small that we can hope to find the winning strategy at the first move, and where our algorithm finds it before MCTS. We believe this is due to the automatic pruning.

1.7. Comparison with AlphaBeta

Assume that ${\mathcal{T}}$ is a finite balanced tree, i.e. that all the nodes with the same depth have the same degree, and that we have some i.i.d. outcomes on the leaves. This slightly generalizes Pearl’s game. Then Tarsi [23] showed that AlphaBeta is optimal in the sense of the expected number of leaves necessary to perfectly find $R(r)$. For such a game, Bruno Scherrer told us that our algorithm visits the leaves precisely in the same order as AlphaBeta, up to some random permutation (see also Subsection 8.11 for a rather convincing numerical indication in this direction). The advantage is that we provide an estimated value during the whole process, while AlphaBeta produces nothing before it really finds $R(r)$.

This strict similarity with AlphaBeta does not hold generally, because our algorithm takes into account the degrees of the nodes. On the one hand, a player is happy to find a node with more possible moves than expected. On the other hand, the value of such a node may be difficult to determine. So the way our algorithm takes degrees into account is complicated and not very transparent.

1.8. Organization of the paper

In the next section, we precisely state our main results and describe our algorithm. Section 3 is devoted to the convergence proof. In Sections 4 and 5, we establish our main result. Section 6 is devoted to the computation of the functions $m$ and $s$ for particular models. In Section 7, we show on an example that global optimality fails. We present numerical experiments in Section 8. In Section 9, we precisely describe the versions of MCTS and its variant we used to test our algorithms.

2. Notation and main results

2.1. Notation

We first introduce once for all the whole notation we need concerning trees.

Let ${\mathbb{T}}$ be the complete discrete ordered tree with root $r$ and infinite degree. An element of ${\mathbb{T}}$ is a finite word composed of letters in ${\mathbb{N}}_{*}$. The root $r$ is the empty word. If e.g. $x=n_{1}n_{2}n_{3}$, this means that $x$ is the $n_{3}$-th child of the $n_{2}$-th child of the $n_{1}$-th child of the root. We consider ordered trees to simplify the presentation, but the order will not play any role.

The depth (or generation) $|x|$ of $x\in{\mathbb{T}}$ is the number of letters of $x$. In particular, $|r|=0$.

We say that $y\in{\mathbb{T}}$ is the father of $x\in{\mathbb{T}}\setminus\{r\}$ (or that $x$ is a child of $y$) if there is $n\in{\mathbb{N}}_{*}$ such that $x=yn$. We denote by $f(x)$ the father of $x$.

For $x\in{\mathbb{T}}$, we call ${\mathbb{C}}_{x}=\{y\in{\mathbb{T}}:f(y)=x\}$ the set of all the children of $x$ and ${\mathbb{T}}_{x}$ the whole progeny of $x$: ${\mathbb{T}}_{x}$ is the subtree of ${\mathbb{T}}$ composed of $x$, its children, its grandchildren, etc.

We say that $x,y\in{\mathbb{T}}$ are brothers if they are different and have the same father. For $x\in{\mathbb{T}}\setminus\{r\}$, we denote by ${\mathbb{H}}_{x}={\mathbb{C}}_{f(x)}\setminus\{x\}$ the set of all the brothers of $x$. Of course, ${\mathbb{H}}_{r}=\emptyset$.

For $x\in{\mathbb{T}}$ and $y\in{\mathbb{T}}_{x}$, we denote by $B_{xy}$ is the branch from $x$ to $y$. In other words, $z\in B_{xy}$ if and only if $z\in{\mathbb{T}}_{x}$ and $y\in{\mathbb{T}}_{z}$.

For $x\in{\mathbb{T}}$, we introduce ${\mathbb{K}}_{x}=\{r\}\cup\bigcup_{y\in B_{rx}\setminus\{x\}}{\mathbb{C}}_{y}=\cup_{y\in B_{rx}}(\{y\}\cup{\mathbb{H}}_{y})$, which consists of $x$, its brothers, its father and uncles, its grandfather and granduncles, etc.

For ${{\mathbf{x}}}\subset{\mathbb{T}}$, we introduce $B_{{\mathbf{x}}}=\cup_{x\in{{\mathbf{x}}}}B_{rx}$, the finite subtree of ${\mathbb{T}}$ with root $r$ and set of leaves ${{\mathbf{x}}}$.

For ${{\mathbf{x}}}\subset{\mathbb{T}}$, we also set ${\mathbb{D}}_{{\mathbf{x}}}=(\cup_{x\in B_{{\mathbf{x}}}}{\mathbb{H}}_{x})\setminus B_{{\mathbf{x}}}$, the set of all the brothers of the elements of $B_{{\mathbf{x}}}$ that do not belong to $B_{{\mathbf{x}}}$. Observe that $B_{{\mathbf{x}}}\cup{\mathbb{D}}_{{\mathbf{x}}}=\cup_{x\in{{\mathbf{x}}}}{\mathbb{K}}_{x}$.

Let ${\mathcal{S}}_{f}$ be the set of all finite subtrees of ${\mathbb{T}}$ with root $r$. For $T$ a finite subset of ${\mathbb{T}}$, it holds that $T\in{\mathcal{S}}_{f}$ if and only if for all $x\in T$, $B_{rx}\subset T$.

For $T\in{\mathcal{S}}_{f}$ and $x\in T$, we introduce $C_{x}^{T}=T\cap{\mathbb{C}}_{x}$ the set of the children of $x$ in $T$, $H^{T}_{x}=T\cap{\mathbb{H}}_{x}$ the set of the brothers of $x$ in $T$, $T_{x}=T\cap{\mathbb{T}}_{x}$ the whole progeny of $x$ in $T$, and $K^{T}_{x}=T\cap{\mathbb{K}}_{x}$ which contains $x$, its brothers (in $T$), its father and uncles (in $T$), its grandfather and granduncles (in $T$), etc. See Figure 1.

We denote by $L_{T}=\{x\in T:T_{x}=\{x\}\}$ the set of the leaves of $T\in{\mathcal{S}}_{f}$. We have $B_{L_{T}}=T$.

Finally, for $T\in{\mathcal{S}}_{f}$ and ${{\mathbf{x}}}\subset T$, we introduce $D_{{\mathbf{x}}}^{T}=T\cap{\mathbb{D}}_{{\mathbf{x}}}$, the set of all the brothers (in $T$) of the elements of $B_{{\mathbf{x}}}$ not belonging to $B_{{\mathbf{x}}}$ (observe that $B_{{\mathbf{x}}}\subset T$), see Figure 2. It holds that $B_{{\mathbf{x}}}\cup D^{T}_{{\mathbf{x}}}=\cup_{x\in{{\mathbf{x}}}}K^{T}_{x}$.

2.2. The general model

We have two players $J_{0}$ and $J_{1}$. The game is modeled by a finite tree ${\mathcal{T}}\in{\mathcal{S}}_{f}$ with root $r$ and leaves ${\mathcal{L}}=L_{\mathcal{T}}$. An element $x\in{\mathcal{T}}$ represents a configuration of the game. On each node $x\in{\mathcal{T}}\setminus{\mathcal{L}}$, we set $t(x)=1$ if it is $J_{1}$’s turn to play when in the configuration $x$ and $t(x)=0$ else. The move is alternate and the player $J_{1}$ starts, so that we have $t(x)={{\mathbf{1}}}_{\{|x|\hbox{ \tiny is even}\}}$, where $|x|$ is the depth of $x$. We have some outcomes $(R(x))_{x\in{\mathcal{L}}}$ in $\{0,1\}$. We say that $x\in{\mathcal{L}}$ is a winning outcome for $J_{1}$ (resp. $J_{0}$) if $R(x)=1$ (resp. $R(x)=0$).

So $J_{1}$ starts, he chooses a node $x_{1}$ among the children of $r$, then $J_{0}$ chooses a node $x_{2}$ among the children of $x_{1}$, and so on, until we reach a leave $y\in{\mathcal{L}}$, and $J_{1}$ is the winner if $R(y)=1$, while $J_{0}$ is the winner if $R(y)=0$.

2.3. Notation

For $x\in{\mathcal{T}}$, we set ${\mathcal{C}}_{x}=C_{x}^{{\mathcal{T}}}$, ${\mathcal{H}}_{x}=H_{x}^{\mathcal{T}}$ and ${\mathcal{K}}_{x}=K_{x}^{\mathcal{T}}$ and, for ${{\mathbf{x}}}\subset{\mathcal{T}}$, ${\mathcal{D}}_{{\mathbf{x}}}=D_{{\mathbf{x}}}^{\mathcal{T}}$.

2.4. Minimax values

Given the whole tree ${\mathcal{T}}$ and the outcomes $(R(x))_{x\in{\mathcal{L}}}$, we can theoretically completely solve the game. We classically define the minimax values $(R(x))_{x\in{\mathcal{T}}}$ as follows. For any $x\in{\mathcal{T}}$, $R(x)=1$ if $J_{1}$ has a winning strategy when starting from $x$ and $R(x)=0$ else (in which case $J_{0}$ necessarily has a winning strategy when starting from $x$).

Remark 1**.**

For $x\in{\mathcal{L}}$, the value of $R(x)$ is prescribed. For $x\in{\mathcal{T}}\setminus{\mathcal{L}}$,

[TABLE]

It is thus possible to compute $R(x)$ for all $x\in{\mathcal{T}}$ by backward induction, starting from the leaves.

This is easily checked: for $x\in{\mathcal{T}}$ with $t(x)=0$, we have $R(x)=0$ if $x$ has at least one child $y\in{\mathcal{T}}$ such that $R(y)=0$ (because $J_{0}$ can choose $y$ from which $J_{1}$ has no winning strategy) and $R(x)=1$ else (because any choice of $J_{0}$ leads to a position $y$ from which $J_{1}$ has a winning strategy). This rewrites $R(x)=\min\{R(y):y\in{\mathcal{C}}_{x}\}$.

If now $t(x)=1$, then $R(x)=1$ if $x$ has at least one child $y\in{\mathcal{T}}$ such that $R(y)=1$ (because $J_{1}$ can choose $y$, from where it has a winning strategy) and $R(x)=0$ else (because any choice of $J_{1}$ leads to a position from which he has no winning strategy). This can be rewritten as $R(x)=\max\{R(y):y\in{\mathcal{C}}_{x}\}$.

2.5. The goal

Our goal is to estimate at best $R(r)$ with a computer and a given amount of time.

In practice, we (say, $J_{1}$) are playing at some true game such as Connect Four or any deterministic game with alternate moves, against a true opponent (say, $J_{0}$). As already mentioned in the introduction, we may always consider that such a game is represented by a tree, and we may remove draws by identifying them to victories of $J_{0}$ (or of $J_{1}$).

We are in some given configuration (after a certain number of true moves of both players). We call this configuration $r$. We have to decide between several possibilities. We thus want to estimate at best from which of these possibilities there is a winning strategy for $J_{1}$. In other words, given a position $r$ (which will be the root of our tree), we want to know at best $R(r)=\max\{R(y):y\in{\mathcal{C}}_{r}\}$: if our estimate suggests that $R(r)=0$, then any move is similarly desperate. If our estimate suggests that $R(r)=1$, this necessarily relies on the fact that we think that some (identified) child $y_{0}$ of $r$ satisfies $R(y_{0})=1$. Then in the true game, we will play $y_{0}$.

Of course, except for very small games, it is not possible in practice to compute $R(r)$ as in Remark 1, because the tree is too large.

The computer knows nothing about the true game except the rules: when it sees a position (node) $x\in{\mathcal{T}}$, it is able to decide if $x$ is terminal position (i.e. $x\in{\mathcal{L}}$) or not; if $x$ is a terminal position, it knows the outcome (i.e. $R(x)$); if $x$ is not a terminal position, it knows who’s turn it is to play (i.e. $t(x)$) and the possible moves (i.e. ${\mathcal{C}}_{x}$).

The true game is deterministic and our study does not apply at all to games of chance such as Backgammon (because games of chance are more difficult to represent as trees, their minimax values or not clearly well-defined, etc.). However, it is a very large game and, in some sense, unknown, so one might hope it resembles the realization of a random model. We will thus assume that ${\mathcal{T}}$, as well as the outcomes $(R(x))_{x\in{\mathcal{L}}}$, are given by the realization of some random model satisfying some independence properties. It is not clear whether such an assumption is reasonable. In any case, our theoretical results completely break down without such a condition.

2.6. A class of algorithms

We consider a large class of algorithms resembling the Monte Carlo Tree Search algorithm, of which a version is recalled in details in the Appendix. The idea is to make $J_{1}$ play against $J_{0}$ a large number of times: the first match is completely random, but then we use the statistics of the preceding matches. MCTS makes $J_{1}$ and $J_{0}$ rather play some moves that often led them to victories. At the end, this provides some ratings for the children of $r$. In the true game, against the true opponent, we then play the move leading to the child of $r$ with the highest rating.

Definition 2**.**

We call a uniformly random match starting from $x\in{\mathcal{T}}$, with $y\in{\mathcal{L}}$ as a resulting leave, the following procedure. Put $y_{0}=x$. If $y_{0}\in{\mathcal{L}}$, set $y=y_{0}$. Else, choose $y_{1}$ uniformly among ${\mathcal{C}}_{y_{0}}$. If $y_{1}\in{\mathcal{L}}$, set $y=y_{1}$. Else, choose $y_{2}$ uniformly among ${\mathcal{C}}_{y_{1}}$. If $y_{2}\in{\mathcal{L}}$, set $y=y_{2}$. Etc. Since ${\mathcal{T}}$ is finite, this always ends up.

The class of algorithms we consider is the following.

Definition 3**.**

An admissible algorithm is a procedure of the following form.

Step 1.* Simulate a uniformly random match from $r$, call $x_{1}$ the resulting leave and set ${{\mathbf{x}}}_{1}=\{x_{1}\}$. Keep track of $R(x_{1})$, of $B_{{{\mathbf{x}}}_{1}}=B_{rx_{1}}$ and of ${\mathcal{D}}_{{{\mathbf{x}}}_{1}}=\cup_{y\in B_{rx_{1}}}{\mathcal{H}}_{y}$.*

Step n+1.* Using only the knowledge of $B_{{{\mathbf{x}}}_{n}}$, ${\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ and $(R(x))_{x\in{{\mathbf{x}}}_{n}}$, choose some (possibly randomized) $z_{n}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\cup{{\mathbf{x}}}_{n}$.*

If $z_{n}\in{{\mathbf{x}}}_{n}$, set $x_{n+1}=z_{n}$, ${{\mathbf{x}}}_{n+1}={{\mathbf{x}}}_{n}$, $B_{{{\mathbf{x}}}_{n+1}}=B_{{{\mathbf{x}}}_{n}}$ and ${\mathcal{D}}_{{{\mathbf{x}}}_{n+1}}={\mathcal{D}}_{{{\mathbf{x}}}_{n}}$.

If $z_{n}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, simulate a uniformly random match starting from $z_{n}$ and call $x_{n+1}$ the resulting leave. Set ${{\mathbf{x}}}_{n+1}={{\mathbf{x}}}_{n}\cup\{x_{n+1}\}$ and keep track of $R(x_{n+1})$, of $B_{{{\mathbf{x}}}_{n+1}}=B_{{{\mathbf{x}}}_{n}}\cup B_{rx_{n+1}}$ and of ${\mathcal{D}}_{{{\mathbf{x}}}_{n+1}}=({\mathcal{D}}_{{{\mathbf{x}}}_{n}}\setminus\{z_{n}\})\cup\bigcup_{y\in B_{z_{n}x_{n+1}}\setminus\{z_{n}\}}{\mathcal{H}}_{y}$.

Conclusion.* Stop after a given number of iterations $n_{0}$ (or after a given amount of time). Choose some (possibly randomized) best child $x_{*}$ of $r$, using only the knowledge of $B_{{{\mathbf{x}}}_{n_{0}}}$, ${\mathcal{D}}_{{{\mathbf{x}}}_{n_{0}}}$ and $(R(x))_{x\in{{\mathbf{x}}}_{n_{0}}}$.*

After $n$ matches, $B_{{{\mathbf{x}}}_{n}}$ represents the explored part of ${\mathcal{T}}$, while ${\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ represents its boundary.

Note that we assume we that know ${\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ (the set of all the brothers, in ${\mathcal{T}}$, of the elements of $B_{{{\mathbf{x}}}_{n}}$ that are not in $B_{{{\mathbf{x}}}_{n}}$) after the simulation some matches leading to the set of leaves ${{\mathbf{x}}}_{n}$. This is motivated by the following reason. Any $y\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ has its father in $B_{{{\mathbf{x}}}_{n}}$. Thus at some point of the simulation, we visited $f(y)$ for the first time and we had to decide (at random) between all its children: we are aware of the fact that $y\in{\mathcal{T}}$.

Also note that we assume that the best thing to do, when visiting a position for the first time (i.e. when arriving at some element of ${\mathcal{D}}_{{{\mathbf{x}}}_{n}}$), is to simulate from there a uniformly random match. This models that fact that we know nothing of the game, except the rules.

The randomization will allow us, in practice, to make some uniform choice in case of equality.

Finally, it seems stupid to allow $x_{n+1}$ to belong to ${{\mathbf{x}}}_{n}$, because this means we will simulate precisely a match we have already simulated: this will not give us some new information. But this avoids many useless discussions. Anyway, a good algorithm will always, or almost always, exclude such a possibility.

Remark 4**.**

(i) In Step n+1, by “using only the knowledge of $B_{{{\mathbf{x}}}_{n}}$, ${\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ and $(R(x))_{x\in{{\mathbf{x}}}_{n}}$ choose some (possibly randomized) $z_{n}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\cup{{\mathbf{x}}}_{n}$”, we mean that $z_{n}=F(B_{{{\mathbf{x}}}_{n}},{\mathcal{D}}_{{{\mathbf{x}}}_{n}},(R(x))_{x\in{{\mathbf{x}}}_{n}},X_{n})$, where $X_{n}\sim{\mathcal{U}}([0,1])$ is independent of everything else and where $F$ is a deterministic measurable application from $A$ to ${\mathbb{T}}$, where $A$ is the set of all $w=(B,D,(\rho(x))_{x\in{{\mathbf{x}}}},u)$, with $B\in{\mathcal{S}}_{f}$, with ${{\mathbf{x}}}=L_{B}$, with $D\subset{\mathbb{D}}_{{{\mathbf{x}}}}$ finite, with $(\rho(x))_{x\in{{\mathbf{x}}}}\in\{0,1\}^{{{\mathbf{x}}}}$, and with $u\in[0,1]$, such that $F(w)\in{{\mathbf{x}}}\cup D$.

(ii) In Conclusion, by “choose some (possibly randomized) best child $x_{*}$ of $r$, using only the knowledge of $B_{{{\mathbf{x}}}_{n_{0}}},{\mathcal{D}}_{{{\mathbf{x}}}_{n_{0}}},(R(x))_{x\in{{\mathbf{x}}}_{n_{0}}}$”, we mean that $x_{*}=G(B_{{{\mathbf{x}}}_{n_{0}}},{\mathcal{D}}_{{{\mathbf{x}}}_{n_{0}}},(R(x))_{x\in{{\mathbf{x}}}_{n_{0}}},X_{n_{0}})$, where $X_{n_{0}}\sim{\mathcal{U}}([0,1])$ is independent of everything else and where $G$ is a deterministic application from $A$ to ${\mathbb{T}}$ such that, for $w\in A$ as above, $G(w)\in C_{r}^{B\cup D}$.

(iii) The two applications $F,G$ completely characterize an admissible algorithm.

2.7. Assumption

Except for the convergence of our class of algorithms, the proof of which being purely deterministic, we will suppose at least the following condition.

Assumption 5**.**

The tree ${\mathcal{T}}$ is a random element of ${\mathcal{S}}_{f}$. We denote by ${\mathcal{L}}=L_{\mathcal{T}}$ and, as already mentioned, we set ${\mathcal{C}}_{x}=C_{x}^{{\mathcal{T}}}$, ${\mathcal{H}}_{x}=H_{x}^{\mathcal{T}}$ and ${\mathcal{K}}_{x}=K_{x}^{\mathcal{T}}$ for $x\in{\mathcal{T}}$ and ${\mathcal{D}}_{{\mathbf{x}}}=D_{{\mathbf{x}}}^{\mathcal{T}}$ for ${{\mathbf{x}}}\subset{\mathcal{T}}$. Conditionally on ${\mathcal{T}}$, we have some random outcomes $(R(x))_{x\in{\mathcal{L}}}$ in $\{0,1\}$. We assume that for any $T\in{\mathcal{S}}_{f}$ with leaves $L_{T}$, the family

[TABLE]

is independent conditionally on $A_{T}=\{T\subset{\mathcal{T}}$ and ${\mathcal{D}}_{L_{T}}=\emptyset\}$ as soon as $\Pr(A_{T})>0$.

Observe that $A_{T}=\{T\subset{\mathcal{T}}$ and $x\in T$ implies ${\mathcal{H}}_{x}\subset T\}=\{T\subset{\mathcal{T}}$ and $\forall\,x\in T$, ${\mathcal{K}}_{x}=K^{T}_{x}\}$.

This condition is a branching property: knowing $A_{T}$, i.e. knowing that $T\subset{\mathcal{T}}$ and that all the brothers (in ${\mathcal{T}}$) of $x\in T$ belong to $T$, we can write ${\mathcal{T}}=T\cup\bigcup_{x\in L_{T}}{\mathcal{T}}_{x}$, and the family $(({\mathcal{T}}_{x},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{x}}),x\in L_{T})$ is independent. A first consequence is as follows.

Remark 6**.**

Suppose Assumption 5. For $T\in{\mathcal{S}}_{f}$ such that $\Pr(A_{T})>0$ and $z\in L_{T}$, we denote by $G_{T,z}$ the law of $({\mathcal{T}}_{z},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{z}})$ conditionally on $A_{T}$. We have $G_{T,z}=G_{K^{T}_{z},z}$.

Indeed, put $S=K^{T}_{z}\subset T$ and observe that $A_{T}=A_{S}\cap\bigcap_{x\in L_{S}\setminus\{z\}}A^{\prime}_{x}$, where we have set $A^{\prime}_{x}=\{T_{x}\subset{\mathcal{T}}_{x},{\mathcal{D}}_{L_{T}}\cap{\mathbb{T}}_{x}=\emptyset\}$. But for each $x\in L_{S}\setminus\{z\}$, $A^{\prime}_{x}\in\sigma({\mathcal{T}}_{x})$. It thus follows from Assumption 5 that $({\mathcal{T}}_{z},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{z}})$ is independent of $\bigcap_{x\in L_{S}\setminus\{z\}}A^{\prime}_{x}$ knowing $A_{S}$. Hence its law knowing $A_{T}$ is the same as knowing $A_{S}$.

Assumption 5 is of course satisfied if ${\mathcal{T}}$ is deterministic and if the family $(R(x))_{x\in{\mathcal{L}}}$ is independent. It also holds true if ${\mathcal{T}}$ is an inhomogeneous Galton-Watson tree and if, conditionally on ${\mathcal{T}}$, the family $(R(x))_{x\in{\mathcal{L}}}$ is independent and (for example) $R(x)$ is Bernoulli with some parameter depending only on the depth $|x|$. But there are many other possibilities, see Section 6 for precise examples of models satisfying Assumption 5.

2.8. Two relevant quantities

Here we introduce two mean quantities necessary to our study.

Definition 7**.**

Suppose Assumption 5. Let $T\in{\mathcal{S}}_{f}$ such that $\Pr(A_{T})>0$, and $z\in L_{T}$. Observe that on $A_{T}$, $z\in{\mathcal{T}}$.

(i) We set $m(T,z)=\Pr(R(z)=1|A_{T})$. By Remark 6, $m(T,z)=m(K^{T}_{z},z)$, because $R(z)$ is of course a deterministic function of $({\mathcal{T}}_{z},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{z}})$, see Remark 1.

(ii) Simulate a uniformly random match starting from $z$, denote by $y$ the resulting leave. We put ${\mathcal{K}}_{zy}={\mathcal{K}}_{y}\cap{\mathbb{T}}_{z}$ and introduce ${\mathcal{G}}=\sigma(y,{\mathcal{K}}_{zy},R(y))$. We set

[TABLE]

By Remark 6, $s(T,z)=s(K^{T}_{z},z)$.

Since $\mathbb{E}[\Pr(R(z)=1|{\mathcal{G}}\lor\sigma(A_{T}))|A_{T}]=m(T,z)$, $s(T,z)$ is a conditional variance.

We will see in Section 6 that for some particular classes of models, $m$ and $s$ can be computed.

Recall that our goal is to produce some admissible algorithm. Assume we have explored $n$ leaves $x_{1},\dots,x_{n}$ and set ${{\mathbf{x}}}_{n}=\{x_{1},\dots,x_{n}\}$. Recall that $B_{{{\mathbf{x}}}_{n}}$ is the explored tree and that ${\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ is, in some sense, its boundary. For $z\in{{\mathbf{x}}}_{n}$, we perfectly know $R(z)$. But for $z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, we only know that $z\in{\mathcal{T}}$ and we precisely know ${\mathcal{K}}_{z}$, since ${\mathcal{K}}_{z}=K^{\mathcal{T}}_{z}=K^{B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}}_{z}$. Thus the best thing we can say is that $R(z)$ equals $1$ with (conditional) probability $m({\mathcal{K}}_{z},z)$. Also, $s({\mathcal{K}}_{z},z)$ quantifies some mean amount of information we will get if handling a uniformly random match starting from $z$.

2.9. The conditional minimax values

From now on, we work with the following setting.

Setting 8**.**

Fix $n\geq 1$. Using an admissible algorithm, we have simulated $n$ matches, leading to the leaves ${{\mathbf{x}}}_{n}=\{x_{1},\dots,x_{n}\}\subset{\mathcal{L}}$. Hence the $\sigma$-field ${\mathcal{F}}_{n}=\sigma({{\mathbf{x}}}_{n},{\mathcal{D}}_{{{\mathbf{x}}}_{n}},(R(x))_{x\in{{\mathbf{x}}}_{n}})$ represents our knowledge of the game. Observe that $B_{{{\mathbf{x}}}_{n}}$ is of course ${\mathcal{F}}_{n}$-measurable. Also, for any $x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, ${\mathcal{K}}_{x}$ and ${\mathcal{H}}_{x}$ are ${\mathcal{F}}_{n}$-measurable, as well as ${\mathcal{C}}_{x}$ if $x\in B_{{{\mathbf{x}}}_{n}}\setminus\{{{\mathbf{x}}}_{n}\}$.

This last assertion easily follows from the fact that for any ${{\mathbf{x}}}\subset{\mathcal{T}}$, any element of $B_{{{\mathbf{x}}}}\cup{\mathcal{D}}_{{{\mathbf{x}}}}$ has all its brothers (in ${\mathcal{T}}$) in $B_{{{\mathbf{x}}}}\cup{\mathcal{D}}_{{{\mathbf{x}}}}$.

We first want to compute $R_{n}(r)=\mathbb{E}[R(r)|{\mathcal{F}}_{n}]=\Pr(R(r)=1|{\mathcal{F}}_{n})$, which is, in some obvious sense, the best approximation of $R(r)$ knowing ${\mathcal{F}}_{n}$. Of course, we will have to compute $R_{n}(x)$ on the whole explored subtree of ${\mathcal{T}}$. We will check the following result in Section 5.

Proposition 9**.**

Grant Assumption 5 and Setting 8. For all $x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, define $R_{n}(x)=\Pr(R(x)=1|{\mathcal{F}}_{n})$. They can be computed by backward induction, starting from ${{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, as follows. For all $x\in{{{\mathbf{x}}}_{n}}$, $R_{n}(x)=R(x)$. For all $x\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, $R_{n}(x)=m({\mathcal{K}}_{x},x)$. For all $x\in B_{{{\mathbf{x}}}_{n}}\setminus{{{\mathbf{x}}}_{n}}$,

[TABLE]

2.10. Main result

We still work under Setting 8. We want to simulate a $(n+1)$-th match. We thus want to choose some $z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ and then simulate a uniformly random match starting from $z$, in such a way that the increase of information concerning $R(r)$ is as large as possible. We unfortunately need a few more notation.

Notation 10**.**

Adopt Setting 8.

(i) For $x\in(B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}})\setminus\{r\}$, we set

[TABLE]

(ii) Define $Z_{n}(x)$, for all $x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, by backward induction, starting from ${{\mathbf{x}}}_{n}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, as follows. If $x\in{{{\mathbf{x}}}_{n}}$, set $Z_{n}(x)=0$. If $x\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, set $Z_{n}(x)=s({\mathcal{K}}_{x},x)$. If $x\in B_{{{\mathbf{x}}}_{n}}\setminus{{\mathbf{x}}}_{n}$, set

[TABLE]

(iii) Fix $z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, handle a uniformly random match starting from $z$, with resulting leave $y_{z}$, set ${{\mathbf{x}}}_{n+1}^{z}={{{\mathbf{x}}}_{n}}\cup\{y_{z}\}$ and denote by ${\mathcal{F}}_{n+1}^{z}=\sigma({{\mathbf{x}}}_{n+1}^{z},{\mathcal{D}}_{{{\mathbf{x}}}_{n+1}^{z}},(R(x))_{{{\mathbf{x}}}_{n+1}^{z}})$ the resulting knowledge. Set $R_{n+1}^{z}(x)=\Pr(R(x)=1|{\mathcal{F}}_{n+1}^{z})$ for all $x\in B_{{{\mathbf{x}}}_{n+1}^{z}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n+1}^{z}}$.

Our main result reads as follows.

Theorem 11**.**

Suppose Assumption 5 and adopt Setting 8 and Notation 10. Define $z_{*}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\cup{{{\mathbf{x}}}_{n}}$ as follows. Put $y_{0}=r$ and set $y_{1}=\arg\!\max\{(U_{n}(y))^{2}Z_{n}(y):y\in{\mathcal{C}}_{y_{0}}\}$. If $y_{1}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\cup{{{\mathbf{x}}}_{n}}$, set $z_{*}=y_{1}$. Else, put $y_{2}=\arg\!\max\{(U_{n}(y))^{2}Z_{n}(y):y\in{\mathcal{C}}_{y_{1}}\}$. If $y_{2}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\cup{{{\mathbf{x}}}_{n}}$, set $z_{*}=y_{2}$. Else, put $y_{3}=\arg\!\max\{(U_{n}(y))^{2}Z_{n}(y):y\in{\mathcal{C}}_{y_{2}}\}$, etc. This procedure necessarily stops since ${\mathcal{T}}$ is finite. Each time we use $\arg\!\max$, we choose e.g. at uniform random in case of equality.

On the event $\{R_{n}(r)\notin\{0,1\}\}$, we have $z_{*}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ and

[TABLE]

Observe that if $R_{n}(r)\in\{0,1\}$, then $R(r)=R_{n}(r)$, because conditionally on ${\mathcal{F}}_{n}$, $R(x)$ is Bernoulli with parameter $R_{n}(r)=\Pr(R(r)=1|{\mathcal{F}}_{n})$. Hence on the event $\{R_{n}(r)\in\{0,1\}\}$, we perfectly know $R(r)$ from ${\mathcal{F}}_{n}$ and thus the $(n+1)$-th match is useless.

When $R_{n}(r)\notin\{0,1\}$, we have the knowledge ${\mathcal{F}}_{n}$, and the theorem tells us how to choose $z_{*}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ such that, after a uniformly random match starting from $z_{*}$, we will estimate at best, in some $L^{2}$-sense, $R(r)$. In words, $z_{*}$ can be found by starting from the root, getting down in the tree $B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ by following the maximum values of $U_{n}^{2}Z_{n}$, until we arrive in ${\mathcal{D}}_{{{\mathbf{x}}}_{n}}$.

As noted by Bruno Scherrer, $z_{*}$ is also optimal if using a $L^{1}$-criterion.

Remark 12**.**

With the assumptions and notation of Theorem 11, it also holds that

[TABLE]

This is easily deduced from (5), noting that conditionally on ${\mathcal{F}}_{n+1}^{z}$ (which contains ${\mathcal{F}}_{n}$), $R(r)$ is Bernoulli$(R_{n+1}^{z}(r))$-distributed, and that for $X\sim$Bernoulli$(p)$, $\mathbb{E}[|X-p|]=2\mathbb{E}[(X-p)^{2}]$.

2.11. The algorithm

The resulting algorithm is as follows.

Algorithm 13**.**

Each time we use $\arg\!\max$, we e.g. choose at uniform random in case of equality.

Step 1.* Simulate a uniformly random match from $r$, call $x_{1}$ the resulting leave and set ${{\mathbf{x}}}_{1}=\{x_{1}\}$.*

During this random match, keep track of $R(x_{1})$, of $B_{{{\mathbf{x}}}_{1}}=B_{rx_{1}}$ and of ${\mathcal{D}}_{{{\mathbf{x}}}_{1}}=\cup_{y\in B_{rx_{1}}}{\mathcal{H}}_{y}$ and set $R_{1}(x)=m({\mathcal{K}}_{x},x)$ and $Z_{1}(x)=s({\mathcal{K}}_{x},x)$ for all $x\in{\mathcal{D}}_{{{\mathbf{x}}}_{1}}$.

Set $x=x_{1}$, $R_{1}(x)=R(x_{1})$ and $Z_{1}(x)=0$.

Do $\{x\!=\!f(x)$, compute $R_{1}(x)$ using (2), $(U_{1}(y))_{y\in{\mathcal{C}}_{x}}$ using (3) and $Z_{1}(x)$ using (4)$\}$ until $x=r$.

Step n+1.* Put $z\!=\!r$. Do $z=\arg\!\max\{(U_{n}(y))^{2}Z_{n}(y):y\in{\mathcal{C}}_{z}\}$ until $z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$. Set $z_{n}=z$.*

Simulate a uniformly random match from $z_{n}$, call $x_{n+1}$ the resulting leave, set ${{\mathbf{x}}}_{n+1}={{\mathbf{x}}}_{n}\cup\{x_{n+1}\}$.

During this random match, keep track of $R(x_{n+1})$, of $B_{{{\mathbf{x}}}_{n+1}}=B_{{{\mathbf{x}}}_{n}}\cup B_{rx_{n+1}}$ and of ${\mathcal{D}}_{{{\mathbf{x}}}_{n+1}}=({\mathcal{D}}_{{{\mathbf{x}}}_{n}}\setminus\{z_{n}\})\cup\bigcup_{y\in B_{z_{n}x_{n+1}}\setminus\{z_{n}\}}{\mathcal{H}}_{y}$ and set $R_{n+1}(x)=m({\mathcal{K}}_{x},x)$ and $Z_{n+1}(x)=s({\mathcal{K}}_{x},x)$ for all $x\in\bigcup_{y\in B_{z_{n}x_{n+1}}\setminus\{z_{n}\}}{\mathcal{H}}_{y}$.

For all $x\in(B_{{{\mathbf{x}}}_{n+1}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n+1}})\setminus B_{rx_{n+1}}$, put $R_{n+1}(x)=R_{n}(x)$ and $Z_{n+1}(x)=Z_{n}(x)$.

For all $x\in(B_{{{\mathbf{x}}}_{n+1}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n+1}})\setminus{\mathcal{K}}_{x_{n+1}}$, put $U_{n+1}(x)=U_{n}(x)$.

Set $x=x_{n+1}$, put $R_{n+1}(x)=R(x_{n+1})$ and $Z_{n+1}(x)=0$.

Do $\{x=f(x)$, compute $R_{n+1}(x)$ using (2), compute $(U_{n+1}(y))_{y\in{\mathcal{C}}_{x}}$ using (3) and $Z_{n+1}(x)$ using (4)$\}$ until $x=r$.

If $R_{n+1}(r)\in\{0,1\}$, go directly to the conclusion.

Conclusion.* Stop after a given number of iterations $n_{0}$ (or after a given amount of time). As best child of $r$, choose $x_{*}=\arg\!\max\{R_{n_{0}}(x):x\in{\mathcal{C}}_{r}\}$.*

2.12. The update is rather quick

For e.g. a (deterministic) regular tree ${\mathcal{T}}$ with degree $d$ and depth $K$, the cost to achieve $n$ steps of the above algorithm is of order $nKd$, because at each step, we have to update the values of $R_{n}(x)$ and $Z_{n}(x)$ for $x\in B_{rx_{n}}$ (which concerns $K$ nodes) and the values of $U_{n}(x)$ for $x\in{\mathcal{K}}_{x_{n+1}}$ (which concerns $Kd$ nodes).

Observe that MCTS algorithms (see Sections 1 and 31) enjoy a cost of order $Kn$, since the updates are done only on the branch $B_{rx_{n}}$ (or even less than that, but in any case we have at least to simulate a random match, of which the cost is proportional to $K$, at each step).

The cost in $Kdn$ for Algorithm 13 seems miraculous. We have not found any deep reason, but calculus, explaining why this theoretically optimal (in a loose sense) behaves so well. It would have been more natural, see Remark 22, that the update would concern the whole explored tree $B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, which contains much more than $Kd$ elements. Very roughly, its cardinal is of order $Kdn$, which would lead to a cost of order $Kdn^{2}$ to achieve $n$ steps.

We did not write it down in the present paper, which is technical enough, but we also studied two variations of Theorem 11.

$\bullet$ First, we considered the very same model, but we tried minimize $\Pr({{\mathbf{1}}}_{\{R_{n+1}^{z}(r)\}>1/2}\neq R(r)|{\mathcal{F}}_{n})$ instead of (5). This is more natural, since in practice, one would rather estimate $R(r)$ by ${{\mathbf{1}}}_{\{R_{n}(r)>1/2\}}$ than by $R_{n}(r)$ (because $R(r)$ takes values in $\{0,1\}$). It is possible to extend our theory, but this leads to an algorithm with a cost of order $Kdn^{2}$ (at least, we found no way to reduce this cost).

$\bullet$ We also studied what happens in the case where the game may lead to draw. Then the outcomes $(A(x))_{x\in{\mathcal{T}}}$ can take three values, [math] (if $J_{0}$ wins), $1$ (if $J_{1}$ wins) and $1/2$ (if the issue is a draw). For any $x\in{\mathcal{T}}$, we can define the minimax rating $A(x)$ as [math] (if $J_{0}$ has a winning strategy), $1$ (if $J_{1}$ has a winning strategy) and $1/2$ (else). The family $(A(x))_{x\in{\mathcal{T}}}$ satisfies the backward induction relation (1). A possibility is to identify a draw to a victory of $J_{0}$ (or of $J_{1}$). Then, under Assumption 5 with $R(x)={{\mathbf{1}}}_{\{A(x)=1\}}$, we can apply directly Theorem 11. However, this leads to an algorithm that tries to find a winning move, but gives up if it thinks it cannot win: the algorithm does not make any difference between a loss and a draw. It is possible to adapt our theory to overcome this default by trying to estimate both $R(x)={{\mathbf{1}}}_{\{A(x)=1\}}$ and $S(x)={{\mathbf{1}}}_{\{A(x)=0\}}$, i.e. to minimize something like $\mathbb{E}[a(R_{n+1}^{z}(r)-R(r))^{2}+b(S_{n+1}^{z}(r)-S(r))^{2}|{\mathcal{F}}_{n}]$ for some $a,b\geq 0$. However, this leads, again, to an algorithm of which the cost is of order $Kdn^{2}$, unless $b=0$ (or $a=0$), which means that we identify a draw to a victory of $J_{0}$ (or of $J_{1}$). Technically, this comes from the fact that in such a framework, nothing like Observation (10) does occur, see also Remark 22.

In practice, one can produce an algorithm taking draws into account as follows: at each step, we compute $(R_{n}(x),Z_{n}(x))_{x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}}$ identifying draws to victories of $J_{0}$ and, with obvious notation, $(R_{n}^{\prime}(x),Z_{n}^{\prime}(x))_{x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}}$ identifying draws to victories of $J_{1}$. We use our algorithm with $(R_{n}(x),Z_{n}(x))_{x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}}$ while $R_{n}(r)$ is not too small, and we then switch to $(R_{n}^{\prime}(x),Z_{n}^{\prime}(x))_{x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}}$. We have no clear idea of how to choose the threshold.

Finally, the situation is even worse for games with a large number (possibly infinite) of game values (representing the gain of $J_{1}$). This could for example be modeled by independent Beta priors on the leaves. As first crippling difficulty, Beta laws are not stable by maximum and minimum.

2.13. Convergence

It is not difficult to check that, even with a completely wrong model, Algorithm 13 always converges in a finite (but likely to be very large) number of steps.

Remark 14**.**

(i) Forgetting everything about theory, we can use Algorithm 13 with any pair of functions $m$ and $s$, both defined on $\{(S,x):S\in{\mathcal{S}}_{f},x\in L_{S}\}$, $m$ being valued in $[0,1]$ and $s$ being valued in $[0,\infty)$.

(ii) For any constant $\lambda>0$, the algorithm using the functions $m$ and $\lambda s$ is precisely the same than the one using $m$ and $s$.

Note that we allow $s$ to be larger than $1$, which is never the case from a theoretical point of view. But in view of (ii), it is very natural. We will prove the following result in Section 3.

Proposition 15**.**

Consider any fixed tree ${\mathcal{T}}\in{\mathcal{S}}_{f}$ with ${\mathcal{L}}$ its set of leaves and any fixed outcomes $(R(x))_{x\in{\mathcal{L}}}$. Denote by $(R(x))_{x\in{\mathcal{T}}}$ the corresponding minimax values. Apply Algorithm 13 with any given pair of functions $m$ and $s$ on $\{(S,x):S\in{\mathcal{S}}_{f},S\subset{\mathcal{T}},x\in L_{S}\}$ with values in $[0,1]$ and $[0,\infty)$ respectively and satisfying the following condition: for any $S\in{\mathcal{S}}_{f}$ with $S\subset{\mathcal{T}}$ and ${\mathcal{D}}_{L_{S}}=\emptyset$, any $x\in L_{S}$, $s(S,x)=0$ if and only if $m(S,x)\in\{0,1\}$, and in such a case, $m(S,x)=R(x)$.

(i) For every $n\geq 1$, $R_{n}(r)\notin\{0,1\}$ implies that $x_{n+1}\notin{{\mathbf{x}}}_{n}$.

(ii) There is $n_{0}\geq 1$ finite such that $R_{n_{0}}(r)\in\{0,1\}$ and we then have $R_{n_{0}}(r)=R(r)$.

Let us emphasize this proposition assumes nothing but the fact that ${\mathcal{T}}$ is finite. Assumption 5 is absolutely not necessary here. The condition on $m$ and $s$ is very general and obviously satisfied if e.g. $m$ is $(0,1)$-valued and if $s$ is $(0,\infty)$-valued.

Once a sufficiently large part of the tree is explored (actually, almost all the tree up to some pruning), the algorithm knows perfectly the minimax value of $r$, even if $m$ and $s$ are meaningless. Thus, the structure of the algorithm looks rather nice. In practice, for a large game, the algorithm will never be able to explore such a large part of the tree, so that the choice of the functions $m$ and $s$ is very important. However, Proposition 15 is reassuring: we hope that even if the modeling is approximate, so that the choices of $m$ and $s$ are not completely convincing, the algorithm might still behave well.

Lemma 16**.**

Under Assumption 5, the functions $m$ and $s$ introduced in Definition 7 satisfy the condition of Proposition 15: for any $S\in{\mathcal{S}}_{f}$ such that $\Pr(A_{S})>0$, any $x\in L_{S}$, $s(S,x)=0$ if and only if $m(S,x)\in\{0,1\}$, and in such a case, $R(x)=m(S,x)$ a.s. on $A_{S}$.

Consequently, we can apply Proposition 15 under Assumption 5 with the functions $m$ and $s$ introduced in Definition 7: for any fixed realization of ${\mathcal{T}}$ and $(R(x))_{x\in{\mathcal{L}}}$, if $S\in{\mathcal{S}}_{f}$ with $S\subset{\mathcal{T}}$ and ${\mathcal{D}}_{L_{S}}=\emptyset$, then $A_{S}$ is realized, so that indeed, for any $x\in L_{S}$, $s(S,x)=0$ if and only if $m(S,x)\in\{0,1\}$, and in such a case, $m(S,x)=R(x)$.

2.14. Practical choice of the functions $m$ and $s$

In Section 6, we will describe a few models satisfying our conditions and for which we can compute the functions $m$ ans $s$. As seen in Definition 7 (see also Algorithm 13), it suffices to be able to compute $m({\mathcal{K}}_{x},x)$ and $s({\mathcal{K}}_{x},x)$ for all $x\in{\mathcal{T}}$ (actually, for all $x$ in the boundary ${\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ of the explored tree). Let us summarize the two main examples.

(i) First, assume that ${\mathcal{T}}$ is an inhomogeneous Galton-Watson tree, with maximum depth $K$ and known reproduction laws, and that conditionally on ${\mathcal{T}}$, the outcomes $(R(x))_{x\in{\mathcal{L}}}$ are independent Bernoulli random variables with parameters depending only on the depths of the involved nodes. Then we will show that the functions $m({\mathcal{K}}_{x},x)$ and $s({\mathcal{K}}_{x},x)$ depend only on the depth of $x$ and can be computed numerically, once for all, from the parameters of the model. See Subsection 6.1 for precise statements. Let us mention that the parameters of the model can be fitted to some real game such as Connect Four (even if it is not clear at all that this model is reasonable) by handling a high number of uniformly random matches, see Subsection 8.2 for a few more explanations.

(ii) Second, assume that ${\mathcal{T}}$ is some given random tree to be specified later. Fix some values $a\in(0,1)$ and $b\in{\mathbb{R}}$. Define $m(\{r\},r)=a$, $s(\{r\},r)=1$ (or any other positive constant, see Remark 14) and, recursively, for all $x\in{\mathcal{T}}$, define $m({\mathcal{K}}_{x},x)$ and $s({\mathcal{K}}_{x},x)$ from $m({\mathcal{K}}_{f(x)},f(x))$, $s({\mathcal{K}}_{f(x)},f(x))$ and $|{\mathcal{C}}_{f(x)}|$ by the formulas

[TABLE]

The formula for $m$ is a modeling symmetry assumption rather well-justified and we can treat the following cases, see in Subsection 6.3 for more details.

(ii)-(a) If we consider Pearl’s model [20], i.e. ${\mathcal{T}}$ is the deterministic $d$-regular tree with depth $K$ and the outcomes are i.i.d. Bernoulli random variables with parameter $p$ (explicit as a function of $a,d,K$), then the above formula for $s$ with $b=2$ is theoretically justified, see Remark 27.

(ii)-(b) Assume next that ${\mathcal{T}}$ is a finite homogeneous Galton-Watson tree with reproduction law $(1-p)\delta_{0}+p\delta_{d}$ (with $pd\leq 1$) and that conditionally on ${\mathcal{T}}$, the outcomes $(R(x))_{x\in{\mathcal{L}}}$ are independent Bernoulli random variables with parameters $(m({\mathcal{K}}_{x},x))_{x\in{\mathcal{L}}}$ (that do not need to be computed). Then if $a=a_{0}$ is well-chosen (as a function of $p$ and $d$), the above formula for $s$ with $b=0$ (i.e. $s\equiv 1$) is theoretically justified, see Remark 29.

We also experimented, without theoretical justification, other values of $b$. But we obtained so few success in this direction that we will not present the corresponding numerical results.

Let us mention that while (i) requires to fit precisely the functions $m$ and $s$ using rough statistics on the true game, (ii) is rather universal. In particular, it seems that the choice $a=0.5$ and $b=0$ works quite well in practice, and this is very satisfying. Also, the implementation is very simple, since each time a new node $x$ is visited by the algorithm, we can compute $m({\mathcal{K}}_{x},x)$ from $m({\mathcal{K}}_{f(x)},f(x))$ and the number of children $|{\mathcal{C}}_{f(x)}|$ of $f(x)$.

Finally observe that for any tree ${\mathcal{T}}$ and any choices of $a\in(0,1)$ and $b\in{\mathbb{R}}$, $m$ is $(0,1)$-valued and $s$ is $(0,\infty)$-valued, so that Proposition 15 applies: the algorithm always converges in a finite number of steps.

2.15. Global optimality fails

By Theorem 11, Algorithm 13 is optimal, in a loose sense, step by step. That is, if knowing, for some $n\geq 1$, ${\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ and the values of $R(x)$ for $x\in{{\mathbf{x}}}_{n}\subset{\mathcal{L}}$, it tells us how to choose the next leave $x_{n+1}\in{\mathcal{L}}$ so that $\mathbb{E}[(R_{n+1}(r)-R(r))^{2}]$ is as small as possible. However, it is not globally optimal.

Remark 17**.**

Let ${\mathcal{T}}$ be the (deterministic) binary tree with depth $3$ and assume that the outcomes $(R(x))_{x\in{\mathcal{L}}}$ are i.i.d. and Bernoulli$(1/2)$-distributed. Then Assumption 5 is satisfied and we can compute the functions $m$ and $s$ introduced in Definition 7. We thus may apply Algorithm 13, producing some random leaves $x_{1},x_{2},\dots$. We set ${\mathcal{F}}_{n}=\sigma(\{B_{{{\mathbf{x}}}_{n}},{\mathcal{D}}_{{{\mathbf{x}}}_{n}},(R(x))_{x\in{{\mathbf{x}}}_{n}}\})$ and $R_{n}(r)=\mathbb{E}[R(r)|{\mathcal{F}}_{n}]$.

There is another admissible algorithm, producing some random leaves ${\tilde{x}}_{1},{\tilde{x}}_{2},\dots$, such that, for $\tilde{\mathcal{F}}_{n}=\sigma(\{B_{{\tilde{\mathbf{x}}}_{n}},{\mathcal{D}}_{{\tilde{\mathbf{x}}}_{n}},(R(x))_{x\in{\tilde{\mathbf{x}}}_{n}}\})$ and ${\tilde{R}}_{n}(r)=\mathbb{E}[R(r)|\tilde{\mathcal{F}}_{n}]$, we have

[TABLE]

It looks very delicate to determine the globally optimal algorithm. Moreover, it is likely that such an algorithm will be very intricate and will not enjoy the quick update property discussed in Subsection 2.12.

3. General convergence

We first show that Algorithm 13 is convergent with any reasonable parameters $m$ and $s$.

Proof of Proposition 15.

We consider some fixed tree ${\mathcal{T}}\in{\mathcal{S}}_{f}$ with ${\mathcal{L}}$ its set of leaves, some fixed outcomes $(R(x))_{x\in{\mathcal{L}}}$ and we denote by $(R(x))_{x\in{\mathcal{T}}}$ the corresponding minimax values. We also consider any pair of functions $m$ and $s$ on $\{(S,x):S\in{\mathcal{S}}_{f},x$ leave of $S\}$ with values in $[0,1]$ and $[0,\infty)$ respectively and we apply Algorithm 13. We assume that for any $S\in{\mathcal{S}}_{f}$ of which $x$ is a leave, $s(S,x)=0$ if and only if $m(S,x)\in\{0,1\}$, and that in such a case, $m(S,x)=R(x)$.

Step 1. After the $n$-th step of the algorithm, we have some values $R_{n}(x)\in[0,1]$, $U_{n}(x)\in[0,1]$ and $Z_{n}(x)\geq 0$ for all $x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, for some ${{\mathbf{x}}}_{n}=\{x_{1},\dots,x_{n}\}\subset{\mathcal{L}}$. These quantities can generally not be interpreted in terms of conditional expectations, but they always obey, by construction, the following rules.

(a) If $x\in{{\mathbf{x}}}_{n}$, then $R_{n}(x)=R(x)$ and $Z_{n}(x)=0$.

(b) If $x\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, then $R_{n}(x)=m({\mathcal{K}}_{x},x)$ and $Z_{n}(x)=s({\mathcal{K}}_{x},x)$.

(c) If $x\in B_{{{\mathbf{x}}}_{n}}\!\setminus{{\mathbf{x}}}_{n}$, $R_{n}(x)={{\mathbf{1}}}_{\{t(x)=0\}}\prod_{y\in{\mathcal{C}}_{x}}R_{n}(y)+{{\mathbf{1}}}_{\{t(x)=1\}}[1-\prod_{y\in{\mathcal{C}}_{x}}(1-R_{n}(y))]$.

(d) If $x\in B_{{{\mathbf{x}}}_{n}}\!\setminus{{\mathbf{x}}}_{n}$, $Z_{n}(x)=\max\{U_{n}^{2}(y)Z_{n}(y):y\in{\mathcal{C}}_{x}\}$.

(e) If $x\in(B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}})\setminus\!\{r\}$, $U_{n}(x)={{\mathbf{1}}}_{\{t(v(x))=0\}}\prod_{y\in{\mathcal{H}}_{x}}R_{n}(y)+{{\mathbf{1}}}_{\{t(v(x))=1\}}\prod_{y\in{\mathcal{H}}_{x}}(1-R_{n}(y)).$

We finally recall that, by Proposition 1,

(f) if $x\in{\mathcal{T}}\setminus{\mathcal{L}}$, $R(x)={{\mathbf{1}}}_{\{t(x)=0\}}\min\{R(y):y\in{\mathcal{C}}_{x}\}+{{\mathbf{1}}}_{\{t(x)=1\}}\max\{R(y):y\in{\mathcal{C}}_{x}\}$.

Step 2. Here we prove that for all $x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$,

[TABLE]

In the whole step, the notions of child (of $x\in B_{{{\mathbf{x}}}_{n}}\setminus{{\mathbf{x}}}_{n}$) and brother (of $x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$) refer to the tree ${\mathcal{T}}$ or, equivalently, to the tree $B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$.

First, (7) is obvious if $x\in{{\mathbf{x}}}_{n}$ by point (a) (then $R_{n}(x)=R(x)\in\{0,1\}$ and $Z_{n}(x)=0$) and if $x\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ by point (b) and our assumption on $m$ and $s$. We next work by backward induction: we consider $x\in B_{{{\mathbf{x}}}_{n}}\setminus{{\mathbf{x}}}_{n}$, we assume that all its children satisfy (7), and we prove that $x$ also satisfies (7). We assume for example that $t(x)=0$, the case where $t(x)=1$ being treated similarly.

If $R_{n}(x)=0$, then by (c), $x$ has (at least) one child $y$ such that $R_{n}(y)=0$ whence, by induction assumption, $R(y)=0$ and thus $R(x)=0$ by (f). Furthermore, $R_{n}(y)=0$ implies that $U_{n}(z)=0$, whence $U_{n}^{2}(z)Z_{n}(z)=0$, for all $z$ brother of $y$ by (e). And by induction assumption, we have $Z_{n}(y)=0$, whence $U_{n}^{2}(y)Z_{n}(y)=0$. We conclude, by (d), that $Z_{n}(x)=0$, and we have seen that $R_{n}(x)=0=R(x)$.

If $R_{n}(x)=1$, then by (c), all the children $y$ of $x$ satisfy $R_{n}(y)=1$, whence, by induction assumption, $R(y)=1$ and thus $R(x)=1$ by (f). Still by induction assumption, $Z_{n}(y)=0$ for all the children $y$ of $x$, whence $Z_{n}(x)=0$ by (d), and we have seen that $R_{n}(x)=1=R(x)$.

Assume now that $Z_{n}(x)=0$, whence $U_{n}^{2}(y)Z_{n}(y)=0$ for all the children $y$ of $x$ by (d). If there is (at least) one child $y$ of $x$ for which $U_{n}(y)=0$, this means that there is another child $z$ of $x$ for which $R_{n}(z)=0$ by (e), whence $R_{n}(x)=0$ by (c). Else, we have $Z_{n}(y)=0$ for all the children $y$ of $x$, so that $R_{n}(y)\in\{0,1\}$ by induction assumption, and thus $R_{n}(x)\in\{0,1\}$ by (c).

We have shown that $R_{n}(x)\in\{0,1\}$ implies $Z_{n}(x)=0$ and $R_{n}(x)=R(x)$, and we have verified that $Z_{n}(x)=0$ implies $R_{n}(x)\in\{0,1\}$. Hence $x$ satisfies (7), which completes the step.

Step 3. We now prove that if $x_{n+1}\in{{\mathbf{x}}}_{n}$, then $R_{n}(r)\in\{0,1\}$ and this will prove point (i). Looking at Algorithm 13, we see that $x_{n+1}\in{{\mathbf{x}}}_{n}$ means that the procedure

[TABLE]

returns some $z\in{{\mathbf{x}}}_{n}$. But then, $Z_{n}(z)=0$ by (a). From (d) and the way $z$ has been built, one easily gets convinced that this implies that $Z_{n}(r)=0$, whence $R_{n}(r)\in\{0,1\}$ by Step 1.

Step 4. By Step 3 and since ${\mathcal{T}}$ has a finite number of leaves, $n_{0}=\inf\{n\geq 1:R_{n}(r)\in\{0,1\}\}$ is well-defined and finite. Finally, we know from Step 2 that $R_{n_{0}}(r)=R(r)$. ∎

4. Preliminaries

We first establish some general formulas concerning the functions $m$ and $s$. They are not really necessary to understand the proof of our main result, but we need them to show Lemma 16. Also, we will use them to derive more tractable expressions in some particular cases in Section 6.

Lemma 18**.**

Suppose Assumption 5 and recall Definition 7. For any $S\in{\mathcal{S}}_{f}$ such that $\Pr(A_{S})>0$ and any $x\in L_{S}$,

[TABLE]

where

[TABLE]

Proof.

We fix $S\in{\mathcal{S}}_{f}$ such that $\Pr(A_{S})>0$ and $x\in L_{S}$. We first observe that for any ${{\mathbf{y}}}\subset{\mathbb{C}}_{x}$ with $|{{\mathbf{y}}}|\geq 1$, $A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}=A_{S\cup{{\mathbf{y}}}}$ (recall that $A_{S}$ is the event on which $S\subset{\mathcal{T}}$ and all the brothers (in ${\mathcal{T}}$) of all the elements of $S$ also belong to $S$).

We now study $m(S,x)=\Pr(R(x)=1|A_{S})$, starting from

[TABLE]

Hence the only difficulty is to verify that $\Pr({\mathcal{C}}_{x}={{\mathbf{y}}},R(x)=1|A_{S})=\Pr({\mathcal{C}}_{x}={{\mathbf{y}}}|A_{S})\Theta(S,x,{{\mathbf{y}}})$ or, equivalently, that

[TABLE]

Since $A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}=A_{S\cup{{\mathbf{y}}}}$ and since ${{\mathbf{y}}}\subset L_{S\cup{{\mathbf{y}}}}$, we know from Assumption 5 that the family $({\mathcal{T}}_{y},(R(u))_{u\in{\mathcal{L}}\cap{\mathcal{T}}_{y}})_{y\in{{\mathbf{y}}}}$ is independent conditionally on $A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}$. Consequently, the family $(R(y))_{y\in{{\mathbf{y}}}}$ is independent conditionally on $A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}$ (because $R(y)$ depends only on ${\mathcal{T}}_{y}$ and $(R(u))_{u\in{\mathcal{L}}\cap{\mathcal{T}}_{y}}$, recall Remark 1). We assume e.g. $t(x)=1$. Since $R(x)=\max\{R(y):y\in{\mathcal{C}}_{x}\}$, we may write

[TABLE]

But for $y\in{{\mathbf{y}}}$, $\Pr(R(y)=0|A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\})=\Pr(R(y)=0|A_{S\cup{{\mathbf{y}}}})=1-m(S\cup{{\mathbf{y}}},y)$, whence (8), because $t(x)=1$.

We next study $s$. Knowing $A_{S}$, we handle a uniformly random match starting from $x$, with resulting leave $v$ and we set ${\mathcal{G}}=\sigma(v,R(v),{\mathcal{K}}_{xv})$, where ${\mathcal{K}}_{xv}={\mathcal{K}}_{v}\cap{\mathbb{T}}_{x}$. We recall that $s(S,x)=\mathbb{E}[(R_{1}(x)-m(S,x))^{2}|A_{S}]$, where $R_{1}(x)=\Pr(R(x)=1|{\mathcal{G}}\lor\sigma(A_{S}))$. If $x\in{\mathcal{L}}$, then $v=x$, whence $R(x)$ is ${\mathcal{G}}$-measurable and thus $R_{1}(x)=R(x)$. Consequently,

[TABLE]

We have $\mathbb{E}[(R(x)-m(S,x))^{2}{{\mathbf{1}}}_{\{x\in{\mathcal{L}}\}}|A_{S}]=\sum_{k\in\{0,1\}}\Pr(x\in{\mathcal{L}},R(x)=k|A_{S})[k-m(S,x)]^{2}$. We thus only have to check that $\mathbb{E}[(R_{1}(x)-m(S,x))^{2}{{\mathbf{1}}}_{\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}}|A_{S}]=\Pr({\mathcal{C}}_{x}={{\mathbf{y}}}|A_{S})\sum_{y\in{{\mathbf{y}}}}|y|^{-1}\Gamma(S,x,{{\mathbf{y}}},y)$ or, equivalently, that $\mathbb{E}[(R_{1}(x)-m(S,x))^{2}|A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}]=\sum_{y\in{{\mathbf{y}}}}|y|^{-1}\Gamma(S,x,{{\mathbf{y}}},y)$.

On $A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}$, let $w$ be the child of $x$ belonging to $B_{xv}$. Since $v$ is obtained by handling a uniformly random match starting from $x$, $\Pr(w=y|A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\})=|{{\mathbf{y}}}|^{-1}$ for all $y\in{{\mathbf{y}}}$. We thus only have to verify that

[TABLE]

But $A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}=A_{S\cup{{\mathbf{y}}}}$, so that, by Assumption 5 (and since the random match is independent of everything else), the family $({\mathcal{T}}_{u},(R(z))_{z\in{\mathcal{L}}\cap{\mathcal{T}}_{u}})_{u\in{{\mathbf{y}}}}$ is independent conditionally on $A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}\cap\{w=y\}$. Hence the family $(R(u))_{u\in{{\mathbf{y}}}\setminus\{y\}}$ is independent and independent of $({\mathcal{T}}_{y},(R(z))_{z\in{\mathcal{L}}\cap{\mathcal{T}}_{y}})$ conditionally on $A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}\cap\{w=y\}$. In particular, $(R(u))_{u\in{{\mathbf{y}}}\setminus\{y\}}$ is independent of ${\mathcal{G}}$ conditionally on $A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}\cap\{w=y\}$.

From now on, we assume e.g. that $t(x)=0$.

We have $R(x)=\min\{R(u):u\in{{\mathbf{y}}}\}=\prod_{u\in{{\mathbf{y}}}}R(u)$ on $\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}$ and we conclude from the above independence property that, conditionally on $A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}\cap\{w=y\}=A_{S\cup{{\mathbf{y}}}}\cap\{w=y\}$,

[TABLE]

But $\Pr(R(u)=1|A_{S\cup{{\mathbf{y}}}})=m(S\cup{{\mathbf{y}}},u)$. Adopting the notation $R_{1}(y)=\Pr(R(y)=1|{\mathcal{G}}\lor\sigma(A_{S\cup{{\mathbf{y}}}}))$, we deduce that $R_{1}(x)=R_{1}(y)\prod_{u\in{{\mathbf{y}}}\setminus\{y\}}m(S\cup{{\mathbf{y}}},u)$ on $A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}\cap\{w=y\}$, whence

[TABLE]

Recall that $t(x)=0$. To conclude that (9) holds true, it only remains to verify that

(a) $\mathbb{E}[(R_{1}(y)-m(S\cup{{\mathbf{y}}},y))^{2}|A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}\cap\{w=y\}]=s(S\cup{{\mathbf{y}}},y)$,

(b) $\mathbb{E}[R_{1}(y)|A_{S}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}\cap\{w=y\}]=m(S\cup{{\mathbf{y}}},y)$.

By definition, we have $s(S\cup{{\mathbf{y}}},y)=\mathbb{E}[(R_{1}(y)-m(S\cup{{\mathbf{y}}},y))^{2}|A_{S\cup{{\mathbf{y}}}}]$ conditionally on $\{w=y\}$, because on $\{w=y\}$, $R_{1}(y)=\Pr(R(y)=1|{\mathcal{G}}\lor\sigma(A_{S\cup{{\mathbf{y}}}}))$ is indeed the conditional probability that $R(y)=1$ knowing the information provided by a uniformly random match starting from $w$ (with resulting leave $v$). Point (a) follows.

For (b), we write

[TABLE]

For the second equality, we used that $\{w=y\}\in{\mathcal{G}}\lor\sigma(A_{S\cup{{\mathbf{y}}}})$. For the third equality, we used that $w$ is of course independent of $R(y)$ knowing $A_{S\cup{{\mathbf{y}}}}$. ∎

We next give the

Proof of Lemma 16.

We fix $S\in{\mathcal{S}}_{f}$ such that $\Pr(A_{S})>0$ and $z\in L_{S}$.

If first $m(S,z)=\Pr(R(z)=1|A_{S})=0$, then of course $R(z)=0$ a.s. on $A_{S}$, whence also $\Pr(R(z)=1|{\mathcal{G}}\lor\sigma(A_{S}))=0$ a.s. on $A_{S}$ (recall Definition 7) and thus $s(S,z)=0$.

Similarly, $m(S,z)=1$ implies that $R(z)=1$ a.s. on $A_{S}$ and that $s(S,z)=0$.

It only remains to prove that $s(S,z)=0$ implies that $m(S,z)\in\{0,1\}$.

If $\Pr(z\in{\mathcal{L}}|A_{S})>0$, then either $\Pr(z\in{\mathcal{L}},R(z)=0|A_{S})>0$ or $\Pr(z\in{\mathcal{L}},R(z)=1|A_{S})>0$. If $s(S,z)=0$, we deduce from Lemma 18 that either $[m(S,z)]^{2}=0$ or $[1-m(S,z)]^{2}=0$, whence $m(S,z)\in\{0,1\}$.

If $\Pr(z\in{\mathcal{L}}|A_{S})=0$, we consider a finite tree $T$ with root $z$ such that $\Pr({\mathcal{T}}_{z}=T|A_{S})>0$. We set $U_{z}=S$ and, for all $x\in T\setminus\{z\}$, $U_{x}=S\cup\bigcup_{y\in B_{zx}\setminus\{x\}}C^{T}_{y}\in{\mathcal{S}}_{f}$. It holds that $x\in L_{U_{x}}$ for all $x\in T$ and, if $x\in T\setminus L_{T}$, $U_{x}\cup C^{T}_{x}=U_{y}$ for all $y\in C^{T}_{x}$.

We now prove by backward induction that for any $x\in T$, $s(U_{x},x)=0$ implies that $m(U_{x},x)\in\{0,1\}$. Applied to $x=z$, this will complete the proof.

If first $x\in L_{T}$, then $\Pr(x\in{\mathcal{L}}|A_{U_{x}})>0$, because $A_{S}\cap\{{\mathcal{T}}_{z}=T\}\subset A_{U_{x}}\cap\{x\in{\mathcal{L}}\}$, because $\Pr({\mathcal{T}}_{z}=T|A_{S})>0$ and because $\Pr(A_{S})>0$. We thus have already seen that $s(U_{x},x)=0$ implies that $m(U_{x},x)\in\{0,1\}$.

If next $x\in T\setminus L_{T}$, we introduce ${{\mathbf{y}}}=C^{T}_{x}$ and we see that $\Pr({\mathcal{C}}_{x}={{\mathbf{y}}}|A_{U_{x}})>0$, because $A_{S}\cap\{{\mathcal{T}}_{z}=T\}\subset A_{U_{x}}\cap\{{\mathcal{C}}_{x}={{\mathbf{y}}}\}$, because $\Pr({\mathcal{T}}_{z}=T|A_{S})>0$ and because $\Pr(A_{S})>0$. We deduce from Lemma 18 that if $s(U_{x},x)=0$, then $\Gamma(U_{x},x,{{\mathbf{y}}},y)=0$ for all $y\in{{\mathbf{y}}}$. If e.g. $t(x)=0$, this implies that for all $y\in{{\mathbf{y}}}$ (recall that $U_{x}\cup{{\mathbf{y}}}=U_{y}$),

[TABLE]

Thus we always have $m(U_{x},x)=\prod_{u\in{{\mathbf{y}}}}m(U_{u},u)$ and either (i) $s(U_{u},u)=0$ for all $u\in{{\mathbf{y}}}$ or (ii) there is $u\in{{\mathbf{y}}}$ such that $m(U_{u},u)=0$. In case (i), we deduce from the induction assumption that $m(U_{u},u)\in\{0,1\}$ for all $u\in{{\mathbf{y}}}$, whence $m(U_{x},x)\in\{0,1\}$. In case (ii), we of course have $m(U_{x},x)=0$. ∎

We next study the information provided by some admissible algorithm. Here, Assumption 5 is not necessary. The following result is intuitively obvious, but we found no short proof.

Lemma 19**.**

Recall Setting 8: an admissible algorithm provided some leaves ${{\mathbf{x}}}_{n}=\{x_{1},\dots,x_{n}\}$ together with the objects ${\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ and $(R(x))_{x\in{{\mathbf{x}}}_{n}}$. For any (deterministic) ${{\mathbf{y}}}_{n}=\{y_{1},\dots,y_{n}\}\subset{\mathbb{T}}$, any $D_{n}\subset{\mathbb{D}}_{{{\mathbf{y}}}_{n}}$ and any $(a(y))_{y\in{{\mathbf{y}}}_{n}}\subset\{0,1\}^{{{\mathbf{y}}}_{n}}$, the law of $({\mathcal{T}},(R(y))_{y\in{\mathcal{L}}})$ knowing

[TABLE]

is the same as knowing

[TABLE]

as soon as $\Pr(A^{\prime}_{n})>0$.

Proof.

We work by induction on $n$.

Step 1. We fix $y_{1}\in{\mathbb{T}}$, we set ${{\mathbf{y}}}_{1}=\{y_{1}\}$, we consider $D_{1}\subset{\mathbb{D}}_{{{\mathbf{y}}}_{1}}$ and $a(y_{1})\in\{0,1\}$. In this step we prove that the law of $({\mathcal{T}},(R(y))_{y\in{\mathcal{L}}})$ knowing $A_{1}=\{x_{1}=y_{1},{\mathcal{D}}_{{{\mathbf{y}}}_{1}}=D_{1},R(y_{1})=a(y_{1})\}$ is the same as knowing $A_{1}^{\prime}=\{y_{1}\in{\mathcal{L}},{\mathcal{D}}_{{{\mathbf{y}}}_{1}}=D_{1},R(y_{1})=a(y_{1})\}$.

To this aim, we consider $T\in{\mathcal{S}}_{f}$ such that $y_{1}\in L_{T}$, $D_{{{\mathbf{y}}}_{1}}^{T}=D_{1}$ and $(\alpha(y))_{y\in L_{T}}\in\{0,1\}^{L_{T}}$ such that $\alpha(y_{1})=a(y_{1})$. We have to check that

[TABLE]

or, equivalently, that $p=q$, where

[TABLE]

Recalling that $x_{1}$ is the leave resulting from a uniformly random match starting from $r$, one easily gets convinced that $p=\Pr(x_{1}=y_{1}|{\mathcal{T}}=T)=\prod_{z\in B_{ry_{1}}\setminus\{y_{1}\}}|C^{T}_{z}|^{-1}$. By the same way, $q=\Pr(x_{1}=y_{1}|y_{1}\in{\mathcal{L}},{\mathcal{D}}_{{{\mathbf{y}}}_{1}}=D_{1})=\prod_{z\in B_{ry_{1}}\setminus\{y_{1}\}}|C^{B_{y_{1}}\cup D_{1}}_{z}|^{-1}$. Since $D_{{{\mathbf{y}}}_{1}}^{T}=D_{1}$, we have $C^{T}_{z}=C^{B_{y_{1}}\cup D_{1}}_{z}$ for all $z\in B_{ry_{1}}\setminus\{y_{1}\}$ and the conclusion follows.

Step 2. Assume that the statement holds true with some $n\geq 1$. Consider some deterministic ${{\mathbf{y}}}_{n+1}=\{y_{1},\dots,y_{n+1}\}\subset{\mathbb{T}}$, $D_{n+1}\subset{\mathbb{D}}_{{{\mathbf{y}}}_{n+1}}$ and $(a(y))_{y\in{{\mathbf{y}}}_{n+1}}\subset\{0,1\}^{{{\mathbf{y}}}_{n+1}}$, as well as the events

[TABLE]

Recall that $x_{n+1}$ is chosen as follows: for some deterministic function $F$ as in Remark 4 and some $X_{n}\sim{\mathcal{U}}([0,1])$ independent of everything else, we set $z_{n}=F({{\mathbf{x}}}_{n},{\mathcal{D}}_{{{\mathbf{x}}}_{n}},(R(x))_{x\in{{\mathbf{x}}}_{n}},X_{n})$, which belongs to ${{\mathbf{x}}}_{n}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$. If $z_{n}\in{{\mathbf{x}}}_{n}$, we set $x_{n+1}=z_{n}$, else, we handle a uniformly random match starting from $z_{n}$ and denote by $x_{n+1}$ the resulting leave.

If $y_{n+1}\in{{\mathbf{y}}}_{n}$, then we have $A_{n+1}=A_{n}\cap\{F({{\mathbf{y}}}_{n},D_{n},(a(x))_{x\in{{\mathbf{y}}}_{n}},X_{n})=y_{n+1}\}$ and $A_{n+1}^{\prime}=A_{n}^{\prime}$, where $D_{n}=D_{n+1}$, where $A_{n}=\{{{\mathbf{x}}}_{n}={{\mathbf{y}}}_{n},{\mathcal{D}}_{{{\mathbf{y}}}_{n}}=D_{n},(R(y))_{y\in{{\mathbf{y}}}_{n}}=(a(y))_{y\in{{\mathbf{y}}}_{n}}\}$ and where ${A_{n}^{\prime}=\{{{\mathbf{y}}}_{n}\subset{\mathcal{L}},{\mathcal{D}}_{{{\mathbf{y}}}_{n}}=D_{n},(R(y))_{y\in{{\mathbf{y}}}_{n}}=(a(y))_{y\in{{\mathbf{y}}}_{n}}\}}$. By induction assumption, we know that the law of $({\mathcal{T}},(R(y))_{y\in{\mathcal{L}}})$ knowing $A_{n}$ is the same as knowing $A_{n}^{\prime}$. Since $X_{n}$ is independent of $({\mathcal{T}},(R(y))_{y\in{\mathcal{L}}}),A_{n}$, the law of $({\mathcal{T}},(R(y))_{y\in{\mathcal{L}}})$ knowing $A_{n+1}$ is the same as knowing $A_{n}$ and thus the same as knowing $A_{n+1}^{\prime}$ (which equals $A_{n}^{\prime}$).

If $y_{n+1}\notin{{\mathbf{y}}}_{n}$, let $x$ be the element of $B_{ry_{n+1}}\cap B_{{{\mathbf{y}}}_{n}}$ the closest to $y_{n+1}$ and let $z_{n}$ be the child of $x$ belonging to $B_{ry_{n+1}}$. We set $D_{n}=(D_{n+1}\setminus{\mathbb{T}}_{z_{n}})\cup\{z_{n}\}$. Then $A_{n+1}=A_{n}\cap B_{1}\cap B_{2}$ and $A_{n+1}^{\prime}=A_{n}^{\prime}\cap B_{2}^{\prime}$, where $A_{n}$ and $A_{n}^{\prime}$ are as in the statement and

[TABLE]

First, since $X_{n}$ is independent of everything else, the law of $({\mathcal{T}},(R(y))_{y\in{\mathcal{L}}})$ knowing $A_{n+1}$ is the same as knowing $A_{n}\cap B_{2}$ (from now on, we take the convention that in $B_{2}$, $x_{n+1}$ is the leave resulting from a uniformly random match starting from $z_{n}$). We thus only have to prove that the law of $({\mathcal{T}},(R(y))_{y\in{\mathcal{L}}})$ knowing $A_{n}\cap B_{2}$ is the same as knowing $A_{n}^{\prime}\cap B_{2}^{\prime}$. Consider $T\in{\mathcal{S}}_{f}$ and $(\alpha(y))_{y\in L_{T}}\in\{0,1\}^{L_{T}}$, such that ${{\mathbf{y}}}_{n+1}\subset L_{T}$, $D^{T}_{{{\mathbf{y}}}_{n+1}}=D_{n+1}$ and $(\alpha(y))_{y\in{{\mathbf{y}}}_{n+1}}=(a(y))_{y\in{{\mathbf{y}}}_{n+1}}$. We have to prove that

[TABLE]

We start from

[TABLE]

thanks to our induction assumption. On the one hand, exactly as in Step 1, we have

[TABLE]

On the other hand,

[TABLE]

Since $C^{T}_{u}=C^{B_{z_{n}y_{n+1}}\cup(D_{n+1}\cap{\mathbb{T}}_{z_{n}})}_{u}$ for all $u\in\setminus\{y_{n+1}\}$ (because $D_{{{\mathbf{y}}}_{n+1}}^{T}=D_{n+1}$), we conclude that

[TABLE]

Since finally $\{{\mathcal{T}}\!=\!T,(R(y))_{y\in L_{T}}\!=\!(\alpha(y))_{y\in L_{T}}\}\subset B_{2}^{\prime}$, we conclude that

[TABLE]

which was our goal. ∎

We deduce the following observation, that is crucial to our study.

Lemma 20**.**

Suppose Assumption 5 and recall Setting 8: an admissible algorithm provided some leaves ${{\mathbf{x}}}_{n}=\{x_{1},\dots,x_{n}\}$ together with the objects ${\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ and $(R(x))_{x\in{{\mathbf{x}}}_{n}}$ and we define ${\mathcal{F}}_{n}=\sigma({{\mathbf{x}}}_{n},{\mathcal{D}}_{{{\mathbf{x}}}_{n}},(R(x))_{x\in{{\mathbf{x}}}_{n}})$. Recall also Remark 6.

(i) Knowing ${\mathcal{F}}_{n}$, for all $x\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, the conditional law of $({\mathcal{T}}_{x},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{x}})$ is $G_{{\mathcal{K}}_{x},x}$.

(ii) Knowing ${\mathcal{F}}_{n}$, for all $x\in B_{{{\mathbf{x}}}_{n}}\setminus{{\mathbf{x}}}_{n}$, the family $(({\mathcal{T}}_{u},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{u}},u\in{\mathcal{C}}_{x})$ is independent.

Recall that for all $x\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, ${\mathcal{K}}_{x}$ is ${\mathcal{F}}_{n}$-measurable and that for all $x\in B_{{{\mathbf{x}}}_{n}}\setminus{{\mathbf{x}}}_{n}$, ${\mathcal{C}}_{x}$ is ${\mathcal{F}}_{n}$-measurable. Hence this statement is meaningful.

Proof.

We observe that ${\mathcal{F}}_{n}$ is generated by the events of the form

[TABLE]

as in Lemma 19. Let $A_{n}^{\prime}=\{{{\mathbf{y}}}_{n}\subset{\mathcal{L}},{\mathcal{D}}_{{{\mathbf{y}}}_{n}}=D_{n},(R(y))_{y\in{{\mathbf{y}}}_{n}}=(a(y))_{y\in{{\mathbf{y}}}_{n}}\}$. We see that on $A_{n}^{\prime}$ (which contains $A_{n}$), ${\mathcal{K}}_{x}=K_{x}^{T}$ for all $x\in D_{n}$ and ${\mathcal{C}}_{x}=C_{x}^{T}$ for all $x\in B_{{{\mathbf{x}}}_{n}}\setminus{{\mathbf{x}}}_{n}$, where $T=B_{{{\mathbf{y}}}_{n}}\cup D_{n}$.

To check (i), it thus suffices to prove that knowing $A_{n}$, for all $x\in D_{n}$, the law of $({\mathcal{T}}_{x},(\!R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{x}}$ is $G_{K^{T}_{x},x}$. We fix $x\in D_{n}$. By Lemma 19, it suffices to verify that the law of $({\mathcal{T}}_{x},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{x}}$ knowing $A_{n}^{\prime}$ is $G_{K^{T}_{x},x}$. Recalling that $A_{K^{T}_{x}}=\{K^{T}_{x}\subset{\mathcal{T}},{\mathcal{D}}_{K^{T}_{x}}=\emptyset\}$, we write $A_{n}^{\prime}=A_{K^{T}_{x}}\cap\bigcap_{u\in L_{K^{T}_{x}}\setminus\{x\}}E_{u}$, where

[TABLE]

By Assumption 5, $({\mathcal{T}}_{x},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{x}}$ is independent of $\bigcap_{u\in L_{K^{T}_{x}}\setminus\{x\}}E_{u}$ knowing $A_{K^{T}_{x}}$. Thus the law of $({\mathcal{T}}_{x},(\!R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{x}}$ knowing $A_{n}^{\prime}$ equals the law of $({\mathcal{T}}_{x},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{x}}$ knowing $A_{K^{T}_{x}}$, which is $G_{K^{T}_{x},x}$ by definition.

For (ii), we show that for any $x\in B_{{{\mathbf{y}}}_{n}}\setminus{{\mathbf{y}}}_{n}$, the family $(({\mathcal{T}}_{u},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{u}},u\in C_{x}^{T})$ is independent conditionally on $A_{n}$, or equivalently, conditionally on $A_{n}^{\prime}$. To this aim, we introduce $S=T\setminus\cup_{y\in C^{T}_{x}}({\mathbb{T}}_{y}\setminus\{y\})$ (i.e. $S$ is the tree $T$ from which we have removed all the subtrees strictly below the children of $x$). We write $A_{n}^{\prime}=A_{S}\cap\bigcap_{u\in L_{S}}E_{u}$ with $E_{u}$ as in the proof of (i). We know from Assumption 5 that the family $(({\mathcal{T}}_{u},(R(y))_{y\in{\mathcal{L}}\cap{\mathbb{T}}_{u}},u\in L_{S})$ is independent conditionally on $A_{S}$. Observing that $E_{u}\in\sigma({\mathcal{T}}_{u},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{u}})$ for all $u\in L_{S}$, we conclude that the family $(({\mathcal{T}}_{u},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{u}},u\in L_{S})$ is independent conditionally on $A_{n}^{\prime}$. (Here we used that if a family of random variables $(X_{i})_{i\in I}$ is independent conditionally on some event $A$ and if we have some events $E_{i}\in\sigma(X_{i})$, for $i\in I$, then the family $(X_{i})_{i\in I}$ is independent conditionally on $A\cap\bigcap_{i\in I}E_{i}$). Since $C^{T}_{x}\subset L_{S}$, the conclusion follows. ∎

5. Proof of the main result

In the whole section, we take Assumption 5 for granted. We first compute the conditional minimax values.

Proof of Proposition 9.

We work under Setting 8. If first $x\in{{\mathbf{x}}}_{n}$, then $R(x)$ is ${\mathcal{F}}_{n}$-measurable, so that $R_{n}(x)=\Pr(R(x)=1|{\mathcal{F}}_{n})=R(x)$.

Next, for $x\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, we know from Lemma 20 that the law of $({\mathcal{T}}_{x},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{x}}$ knowing ${\mathcal{F}}_{n}$ is $G_{{\mathcal{K}}_{x},x}$. Recalling Definition 7 and Remark 6, we see that $R_{n}(x)=\Pr(R(x)=1|{\mathcal{F}}_{n})=m({\mathcal{K}}_{x},x)$.

Finally, for $x\in B_{{{\mathbf{x}}}_{n}}\setminus{{\mathbf{x}}}_{n}$, Lemma 20 tells us that the family $(({\mathcal{T}}_{y},(R(u))_{u\in{\mathcal{L}}\cap{\mathcal{T}}_{y}},y\in{\mathcal{C}}_{x})$ is independent conditionally on ${\mathcal{F}}_{n}$. But for $y\in{\mathcal{C}}_{x}$, $R(y)$ is of course $\sigma({\mathcal{T}}_{y},(R(u))_{u\in{\mathcal{L}}\cap{\mathcal{T}}_{y}})$-measurable (recall Remark 1). Thus the family $(R(y),y\in{\mathcal{C}}_{x})$ is independent conditionally on ${\mathcal{F}}_{n}$. If $t(x)=0$, we may write, by (1),

[TABLE]

which equals $\prod_{y\in{\mathcal{C}}_{x}}R_{n}(y)$ as desired. If now $t(x)=1$, we find similarly

[TABLE]

which is nothing but $1-\prod_{y\in{\mathcal{C}}_{x}}(1-R_{n}(y))$. ∎

We now study the different possibilities for the $(n+1)$-th step.

Proposition 21**.**

Adopt Setting 8 and Notation 10. For $z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ and $x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, we set

[TABLE]

(i) We have $\arg\!\min\{\mathbb{E}[(R_{n+1}^{z}(r)-R(r))^{2}|{\mathcal{F}}_{n}]:z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\}=\arg\!\max\{\Delta_{n}^{z}(r):z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\}$.

(ii) For all $z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, we have $\Delta_{n}^{z}(z)=s({\mathcal{K}}_{z},z)$.

(iii) For all $z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, all $x\in B_{rz}\setminus\{r\}$, we have $\Delta_{n}^{z}(f(x))=(U_{n}(x))^{2}\Delta_{n}^{z}(x)$.

(iv) For all $z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, $\Delta_{n}^{z}(r)=s({\mathcal{K}}_{z},z)\prod_{y\in B_{rz}\setminus\{r\}}(U_{n}(y))^{2}$.

Proof.

Point (i) is not difficult: for all $z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$,

[TABLE]

Using that ${\mathcal{F}}_{n}\subset{\mathcal{F}}_{n+1}^{z}$ and that $R_{n+1}^{z}(r)=\mathbb{E}[R(r)|{\mathcal{F}}_{n+1}^{z}]$, we conclude that

[TABLE]

Since the left hand side does not depend on $z$, minimizing $\mathbb{E}[(R_{n+1}^{z}(r)-R(r))^{2}|{\mathcal{F}}_{n}]$ is equivalent to maximizing $\mathbb{E}[(R_{n+1}^{z}(r)-R_{n}(r))^{2}|{\mathcal{F}}_{n}]$.

Also, point (iv) immediately follows from points (ii) and (iii).

To check points (ii) and (iii), we fix $z\in{\mathcal{D}}_{x_{n}}$. We recall that ${\mathcal{F}}_{n+1}^{z}={\mathcal{F}}_{n}\lor{\mathcal{G}}$, where ${\mathcal{G}}=\sigma(y,{\mathcal{K}}_{zy},R(y))$, where $y$ is the leave resulting from a uniformly random match starting from $z$ and where ${\mathcal{K}}_{zy}={\mathcal{K}}_{y}\cap{\mathbb{T}}_{z}$. We also recall that $R_{n+1}^{z}(x)=\Pr(R(x)=1|{\mathcal{F}}_{n+1}^{z})$ for all $x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$.

We know from Lemma 20 that the law of $({\mathcal{T}}_{z},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{z}})$ knowing ${\mathcal{F}}_{n}$ is $G_{{\mathcal{K}}_{z},z}$. Recalling Definition 7 and Remark 6, we immediately deduce that $R_{n}(z)=m({\mathcal{K}}_{z},z)$ and that

[TABLE]

This proves (ii). To prove (iii), we fix $x\in B_{rz}\setminus\{r\}$ and we set $v=f(x)$. By Lemma 20, the family $(({\mathcal{T}}_{y},(R(u))_{u\in{\mathcal{L}}\cap{\mathcal{T}}_{y}}),y\in{\mathcal{C}}_{v})$ is independent conditionally on ${\mathcal{F}}_{n}$. Furthermore, ${\mathcal{G}}$, which only concerns $({\mathcal{T}}_{x},(R(u))_{u\in{\mathcal{L}}\cap{\mathcal{T}}_{x}})$ is independent of the family $(({\mathcal{T}}_{y},(R(u))_{u\in{\mathcal{L}}\cap{\mathcal{T}}_{y}}),y\in{\mathcal{H}}_{x})$. Finally, we recall that for all $y\in{\mathcal{C}}_{v}$, $R(y)$ is $\sigma({\mathcal{T}}_{y},(R(u))_{u\in{\mathcal{L}}\cap{\mathcal{T}}_{y}})$-measurable.

If $t(v)=0$,

[TABLE]

whence, by conditional independence, $R_{n+1}^{z}(v)=\prod_{y\in{\mathcal{C}}_{v}}\Pr(R(y)=1|{\mathcal{F}}_{n}\lor{\mathcal{G}})$ and thus

[TABLE]

whence $R_{n+1}^{z}(v)=U_{n}(x)R_{n+1}^{z}(x)$. Since now $R_{n}(v)=\prod_{y\in{\mathcal{C}}_{v}}R_{n}(y)=U_{n}(x)R_{n}(x)$ by Proposition 9, we conclude that $(R_{n+1}(v)-R_{n}(v))^{2}=U_{n}^{2}(x)(R_{n+1}^{z}(x)-R_{n}(x))^{2}$, whence, taking expectations conditionally on ${\mathcal{F}}_{n}$, $\Delta^{z}_{n}(v)=U_{n}^{2}(x)\Delta^{z}_{n}(x)$.

If next $t(v)=1$,

[TABLE]

so that $R_{n+1}^{z}(v)=1-\prod_{y\in{\mathcal{C}}_{v}}\Pr(R(y)=0|{\mathcal{F}}_{n}\lor{\mathcal{G}})$. We conclude that

[TABLE]

so that $R^{z}_{n+1}(v)=1-U_{n}(x)(1-R_{n+1}^{z}(x))$. Since finally $R_{n}(v)=1-\prod_{y\in{\mathcal{C}}_{v}}(1-R_{n}(y))=1-U_{n}(x)(1-R_{n}(x))$ by Proposition 9, $(R_{n+1}^{z}(v)-R_{n}(v))^{2}=U_{n}^{2}(x)(R_{n+1}^{z}(x)-R_{n}(x))^{2}$, whence, taking expectations conditionally on ${\mathcal{F}}_{n}$, $\Delta^{z}_{n}(v)=U_{n}^{2}(x)\Delta^{z}_{n}(x)$. ∎

We now have all the weapons to give the

Proof of Theorem 11.

Recall that we work under Setting 8 and that we adopt Notation 10 in which $U_{n}$, $Z_{n}$ and ${\mathcal{F}}^{n+1}_{z}$ are defined. For $x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, we set $\bar{U}_{n}(x)=\prod_{B_{rx}\setminus\{r\}}U_{n}(y)$, with the convention that $\bar{U}_{n}(r)=1$.

Step 1. In view of the explicit formula for $\Delta_{n}^{z}(r)$ checked in Proposition 21-(iv), the natural way to find $z_{*}$ maximizing $\Delta_{n}^{z}(r)$ is to start from $r$ and to go down in $B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ following the maximum values of $(N_{n}(x))_{x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}}$ defined as follows. Set $N_{n}(x)=0$ for $x\in{{\mathbf{x}}}_{n}$, set $N_{n}(x)=(\bar{U}_{n}(x))^{2}s({\mathcal{K}}_{x},x)$ for $x\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ and put $N_{n}(x)=\max\{N_{n}(y):y\in{\mathcal{C}}_{x}\}$ for $x\in B_{{{\mathbf{x}}}_{n}}\setminus{{\mathbf{x}}}_{n}$.

We claim that $N_{n}(x)=(\bar{U}_{n}(x))^{2}Z_{n}(x)$ for all $x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$.

Indeed, set ${\tilde{N}}_{n}(x)=(\bar{U}_{n}(x))^{2}Z_{n}(x)$ and recall Notation 10-(ii). We obviously have ${\tilde{N}}_{n}(x)=N_{n}(x)$ for $x\in{{\mathbf{x}}}_{n}$ (because then $Z_{n}(x)=0$) and for $x\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ (because then $Z_{n}(x)=s({\mathcal{K}}_{x},x)$). And for $x\in B_{{{\mathbf{x}}}_{n}}\setminus{{\mathbf{x}}}_{n}$, we have ${\tilde{N}}_{n}(x)=(\bar{U}_{n}(x))^{2}\max\{(U_{n}(y))^{2}Z_{n}(y):y\in{\mathcal{C}}_{x}\}=\max\{{\tilde{N}}_{n}(y):y\in{\mathcal{C}}_{x}\}$, because for $y\in{\mathcal{C}}_{x}$, we have $\bar{U}_{n}(x)U_{n}(y)=\bar{U}_{n}(y)$. The claim follows by (backward) induction.

Step 2. We define $z^{*}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\cup{{\mathbf{x}}}_{n}$ as follows. Put $y_{0}=r$. Find $y_{1}=\arg\!\max\{N_{n}(y):y\in{\mathcal{C}}_{y_{0}}\}$. If $y_{1}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\cup{{\mathbf{x}}}_{n}$, set $z^{*}=y_{1}$. Else, put $y_{2}=\arg\!\max\{N_{n}(y):y\in{\mathcal{C}}_{y_{1}}\}$. If $y_{2}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\cup{{\mathbf{x}}}_{n}$, set $z^{*}=y_{2}$. Else, put $y_{3}=\arg\!\max\{N_{n}(y):y\in{\mathcal{C}}_{y_{2}}\}$, etc.

By construction, $z^{*}=\arg\!\max\{N_{n}(z):z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\cup{{\mathbf{x}}}_{n}\}$. Also, $N_{n}(x)=N_{n}(r)$ for all $x\in B_{rz^{*}}$.

Step 3. We recall that $z_{*}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\cup{{\mathbf{x}}}_{n}$ was defined, similarly, as follows: put $y_{0}=r$ and find $y_{1}=\arg\!\max\{U_{n}^{2}(y)Z_{n}(y):y\in{\mathcal{C}}_{y_{0}}\}$. If $y_{1}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\cup{{\mathbf{x}}}_{n}$, set $z_{*}=y_{1}$. Else, put $y_{2}=\arg\!\max\{U_{n}^{2}(y)Z_{n}(y):y\in{\mathcal{C}}_{y_{1}}\}$. If $y_{2}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\cup{{\mathbf{x}}}_{n}$, set $z_{*}=y_{2}$. Else, put $y_{3}=\arg\!\max\{U_{n}^{2}(y)Z_{n}(y):y\in{\mathcal{C}}_{y_{2}}\}$, etc.

Step 4. We now prove that if $N_{n}(r)>0$, then $z_{*}=z^{*}$.

First observe that for any $x\in B_{{{\mathbf{x}}}_{n}}\setminus{{\mathbf{x}}}_{n}$ such that $\bar{U}_{n}(x)>0$,

[TABLE]

Furthermore, we have $N_{n}(x)=N_{n}(r)$ for all $x\in B_{rz^{*}}$. Hence if $N_{n}(r)>0$, then $\bar{U}_{n}(x)>0$ for all $x\in B_{rz^{*}}$ (recall that $N_{n}(x)=(\bar{U}_{n}(x))^{2}Z_{n}(x)$). Consequently, (10) holds true during the whole computation of $z^{*}$, so that $z_{*}=z^{*}$.

Step 5. By Steps 2 and 4, $z_{*}=z^{*}=\arg\!\max\{N_{n}(z):z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\cup{{\mathbf{x}}}_{n}\}$ on $\{N_{n}(r)>0\}$. And we know from Proposition 15 and Lemma 16 that on $\{R_{n}(r)\notin\{0,1\}\}$, $z_{*}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$. We also know that $R_{n}(r)\notin\{0,1\}$ implies that $Z_{n}(r)>0$ (see Step 2 of the proof of Proposition 15), whence $N_{n}(r)=Z_{n}(r)>0$ (recall Step 1 and that $\bar{U}_{n}(r)=1$). Thus on $\{R_{n}(r)\notin\{0,1\}\}$, we have $z_{*}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ and $z_{*}=\arg\!\max\{N_{n}(z):z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\}=\arg\!\max\{(\bar{U}_{n}(z))^{2}s({\mathcal{K}}_{z},z):z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\}$ by definition of $N_{n}$, see Step 1. By Proposition 21-(iv), we conclude that on $\{R_{n}(r)\notin\{0,1\}\}$

[TABLE]

which equals $\arg\!\min\{\mathbb{E}[(R_{n+1}^{z}(r)-R(r))^{2}|{\mathcal{F}}_{n}]:z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\}$ by Proposition 21-(i). We have verified that on $\{R_{n}(r)\notin\{0,1\}\}$, $z_{*}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ and $z_{*}=\arg\!\min\{\mathbb{E}[(R_{n+1}^{z}(r)-R(r))^{2}|{\mathcal{F}}_{n}]:z\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}\}$, which was our goal. ∎

Remark 22**.**

As seen in the proof, the natural way to find $z_{*}$ would be to start from $r$ and to go down in the tree following the highest values of $N_{n}$. Recalling the discussion of Subsection 2.12, this would lead to an algorithm with cost of order $Kdn^{2}$: since (generally) $R_{n+1}(x)\neq R_{n}(x)$ for $x\in B_{rx_{n+1}}$, this (generally) modifies the value of $\bar{U}_{n}(x)$ for all $x\in B_{{{\mathbf{x}}}_{n}}\setminus B_{rx_{n+1}}$ (actually, except for $x\in B_{{{\mathbf{x}}}_{n}}\cap B_{rx_{n+1}}$) and thus the values of $N_{n}(x)$ for all $x$ on the whole explored tree $B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$. The observation (10), which asserts that $\arg\!\max\{N_{n}(y):y\in{\mathcal{C}}_{x}\}=\arg\!\max\{(U_{n}(y))^{2}Z_{n}(y):y\in{\mathcal{C}}_{x}\}$ is thus crucial, as well as the fact that $U_{n}$ and $Z_{n}$ enjoy a quick update property.

6. Computation of the parameters for a few specific models

Here we present a few models for the tree ${\mathcal{T}}$ and the outcomes $(R(x))_{x\in{\mathcal{L}}}$ where our assumptions are met and where we can compute, at least numerically, the functions $m$ and $s$ introduced in Definition 7. Recall that these functions are necessary to implement Algorithm 13.

6.1. Inhomogeneous Galton-Watson trees

We assume that ${\mathcal{T}}$ is the realization of an inhomogeneous Galton-Watson tree with reproduction laws $\mu_{0},\dots,\mu_{K}$: the number of children of the root $r$ follows the law $\mu_{0}\in{\mathcal{P}}({\mathbb{N}})$, the number of children of these children are independent and $\mu_{1}$-distributed, etc. We assume that $\mu_{K}=\delta_{0}$, so that any individual of generation $K$ is a leave and thus $K$ is the maximal depth of ${\mathcal{T}}$.

We also consider a family $q_{0},\dots,q_{K}$ of numbers in $[0,1]$. Conditionally on ${\mathcal{T}}$, we assume that the family $(R(x))_{x\in{\mathcal{L}}}$ is independent and that $R(x)\sim$ Bernoulli$(q_{|x|})$ for all $x\in{\mathcal{L}}$.

Example 23**.**

With such a model, Assumption 5 is fulfilled, and for any $S\in S_{f}$ and $x\in L_{S}$ such that $\Pr(A_{S})>0$, we have $m(S,x)={\mathfrak{m}}(|x|)$ and $s(S,x)={\mathfrak{s}}(|x|)$, where

(i) ${\mathfrak{m}}$ is defined by backward induction by ${\mathfrak{m}}(K)=q_{K}$ and, for $k=0,\dots,K-1$,

[TABLE]

(ii) ${\mathfrak{s}}$ is defined by backward induction by ${\mathfrak{s}}(K)=q_{K}(1-q_{K})$ and, for $k=0,\dots,K-1$,

[TABLE]

These quantities can be computed once for all if one knows the parameters $\mu_{0},\dots,\mu_{K}$ and $q_{0},\dots,q_{K}$ of the model. If unknown, as is generally the case, these parameters can be evaluated numerically quite precisely, handling an important number of uniformly random matches. From these evaluations, we can derive some approximations of ${\mathfrak{m}}$ and ${\mathfrak{s}}$. However, the main problem is of course that in general, assuming that the true game is the realization of such a model is not very realistic.

Proof.

First, Assumption 5 is satisfied, thanks to the classical branching property of Galton-Watson trees. Indeed, consider $S\in{\mathcal{S}}_{f}$ and $x\in L_{S}$ such that $\Pr(A_{S})>0$. Conditionally on $A_{S}$, we can write ${\mathcal{T}}=S\cup\bigcup_{x\in L_{S}}{\mathcal{T}}_{x}$ and the family $(({\mathcal{T}}_{x},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{x}},x\in L_{S})$ is independent by construction. Furthermore, for any $x\in L_{S}$, the law $G_{S,x}$ of $({\mathcal{T}}_{x},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{x}}$ knowing $A_{S}$ depends only on the depth $|x|$.

Consequently, there are $({\mathfrak{m}}(k))_{k=0,\dots,K}$ and $({\mathfrak{s}}(k))_{k=0,\dots,K}$ such that for $S\in{\mathcal{S}}_{f}$ and $x\in L_{S}$ with $\Pr(A_{S})>0$, $m(S,x)={\mathfrak{m}}(|x|)$ and $s(S,x)={\mathfrak{s}}(|x|)$.

If $|x|=K$, then $x$ is necessarily a leave, so that $\Pr(x\in{\mathcal{L}}|A_{S})=1$, whence, by Lemma 18, ${\mathfrak{m}}(K)=m(S,x)=\Pr(x\in{\mathcal{L}},R(x)=1|A_{S})=q_{K}$ and

[TABLE]

which equals $q_{K}(1-q_{K})^{2}+(1-q_{K})q_{K}^{2}=q_{K}(1-q_{K})$ as desired.

Finally, the proof can be completed by using Lemma 18 and that if $|x|=k\in\{0,\dots,K-1\}$ and if for example $t(x)=0$ (i.e. $k$ is odd), for any $\ell\geq 1$,

$\bullet$ $\Pr(x\in{\mathcal{L}},R(x)=1|A_{S})=\mu_{k}(0)q_{k}$ and $\Pr(x\in{\mathcal{L}},R(x)=0|A_{S})=\mu_{k}(0)(1-q_{k})$,

$\bullet$ $\sum_{{{\mathbf{y}}}\subset{\mathbb{C}}_{x},|{{\mathbf{y}}}|=\ell}\Pr({\mathcal{C}}_{x}={{\mathbf{y}}}|A_{S})=\mu_{k}(\ell)$,

$\bullet$ $\Theta(S,x,{{\mathbf{y}}})$ and $\Gamma(S,x,{{\mathbf{y}}},y)$ depend only on $k$ and on $\ell=|{{\mathbf{y}}}|$ and (since $t(x)=0$), $\Theta(S,x,{{\mathbf{y}}})=[{\mathfrak{m}}(k+1)]^{\ell}$ and $\Gamma(S,x,{{\mathbf{y}}},y)={\mathfrak{s}}(k+1)[{\mathfrak{m}}(k+1)]^{2\ell-2}+[({\mathfrak{m}}(k+1))^{\ell}-{\mathfrak{m}}(k)]^{2}$. ∎

Remark 24**.**

In Pearl’s model [20], ${\mathcal{T}}$ is the deterministic regular tree with degree $d\geq 2$ and depth $K\geq 1$ and the family $(R(x))_{x\in{\mathcal{L}}}$ is i.i.d. Bernoulli$(p)$-distributed. This is a particular case of Example 23 with $\mu_{0}=\dots=\mu_{K-1}=\delta_{d}$ and $\mu_{K}=\delta_{0}$ and $q_{K}=p$ (the values of $(q_{k})_{k=0,\dots,K-1}$ being irrelevant). One thus finds ${\mathfrak{m}}(K)=p$, ${\mathfrak{s}}(K)=p(1-p)$ and, for $k=0,\dots,K-1$,

[TABLE]

6.2. Inhomogeneous Galton-Watson trees of order two

Here we mention that we can also deal with random trees that enjoy some independence properties without being Galton-Watson trees. For example, the following model of order $2$ allows one to build a broad class of random trees with non-increasing degree (along each branch), which might be useful for real games. It is possible to treat some models of higher order, but the functions $m$ and $s$ then become really tedious to compute theoretically and to approximate in practice.

We consider a family of probability measures on ${\mathbb{N}}$: $\mu_{0}$ and $\mu_{k,d}$ for $k=1,\dots,K$ and $d\geq 1$. We assume that $\mu_{K,d}=\delta_{0}$ for all $d\geq 1$ and $K$ will represent the maximum depth of the tree.

We build the random tree ${\mathcal{T}}$ as follows: the root has $D_{r}\sim\mu_{0}$ children. Conditionally on $D_{r}$, all the children $x$ of the root produce, independently, a number $D_{x}\sim\mu_{1,D_{r}}$ of children. Once everything is built up to generation $k\in\{0,\dots,K-1\}$, all the individuals $x$ with $|x|=k$ produce, independently (conditionally on what is already built), a number $D_{x}\sim\mu_{k,D_{f(x)}}$ of children.

We also consider a family $q_{0},\dots,q_{K}$ of numbers in $[0,1]$. Conditionally on ${\mathcal{T}}$, we assume that the family $(R(x))_{x\in{\mathcal{L}}}$ is independent and that $R(x)\sim$ Bernoulli$(q_{|x|})$ for all $x\in{\mathcal{L}}$.

Example 25**.**

With such a model, Assumption 5 is fulfilled, and for any $S\in S_{f}$ (with $\{r\}\subsetneq S$) such that $\Pr(A_{S})>0$ and any $x\in L_{S}$, we have we have $m(S,x)={\mathfrak{m}}(|x|,|C^{S}_{f(x)}|)$ and $s(S,x)={\mathfrak{s}}(|x|,|C^{S}_{f(x)}|)$, where ${\mathfrak{m}}$ and ${\mathfrak{s}}$ can be computed by backward induction as follows.

(i) ${\mathfrak{m}}(K,d)=q_{K}$ for all $d\geq 1$ and, for $k=1,\dots,K-1$ and $d\geq 1$,

[TABLE]

(ii) ${\mathfrak{s}}(K,d)=q_{K}(1-q_{K})$ for all $d\geq 1$ and, for $k=1,\dots,K-1$ and $d\geq 1$,

[TABLE]

We could easily express $m(\{r\},r)$ and $s(\{r\},r)$, but these values are useless as far as Algorithm 13 is concerned.

Proof.

First, Assumption 5 is satisfied. Indeed, consider $S\in{\mathcal{S}}_{f}$ and $x\in L_{S}$ such that $\Pr(A_{S})>0$. Conditionally on $A_{S}$, we can write ${\mathcal{T}}=S\cup\bigcup_{x\in L_{S}}{\mathcal{T}}_{x}$ and the family $(({\mathcal{T}}_{x},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{x}},x\in L_{S})$ is independent by construction. Furthermore, for any $x\in L_{S}$, the law $G_{S,x}$ of $({\mathcal{T}}_{x},(R(y))_{y\in{\mathcal{L}}\cap{\mathcal{T}}_{x}}$ knowing $A_{S}$ depends only on the depth $|x|$ and of $|C_{f(x)}^{S}|$ (except if $S=\{r\}$ and $x=r$).

Consequently, there are $({\mathfrak{m}}(k,d))_{k=1,\dots,K,d\geq 1}$ and $({\mathfrak{s}}(k,d))_{k=1,\dots,K,d\geq 1}$ such that for $S\in{\mathcal{S}}_{f}$ and $x\in L_{S}$ with $|C^{S}_{f(x)}|=d$, $\Pr(A_{S})>0$, $m(S,x)={\mathfrak{m}}(|x|,d)$ and $s(S,x)={\mathfrak{s}}(|x|,d)$.

If $|x|=K$, then $x$ is necessarily a leave, so that $\Pr(x\in{\mathcal{L}}|A_{S})=1$, whence, by Lemma 18, ${\mathfrak{m}}(K,d)=m(S,x)=\Pr(x\in{\mathcal{L}},R(x)=1|A_{S})=q_{K}$ and

[TABLE]

which equals $q_{K}(1-q_{K})^{2}+(1-q_{K})q_{K}^{2}=q_{K}(1-q_{K})$ as desired.

Finally, the proof can be completed by using Lemma 18 and that if $|x|=k\in\{0,\dots,K-1\}$ (and $|C^{S}_{f(x)}|=d$) and if for example $t(x)=0$ (i.e. $k$ is odd), for any $\ell\geq 1$,

$\bullet$ $\Pr(x\in{\mathcal{L}},R(x)=1|A_{S})=\mu_{k,d}(0)q_{k}$ and $\Pr(x\in{\mathcal{L}},R(x)=0|A_{S})=\mu_{k,d}(0)(1-q_{k})$,

$\bullet$ $\sum_{{{\mathbf{y}}}\subset{\mathbb{C}}_{x},|{{\mathbf{y}}}|=\ell}\Pr({\mathcal{C}}_{x}={{\mathbf{y}}}|A_{S})=\mu_{k,d}(\ell)$,

$\bullet$ $\Theta(S,x,{{\mathbf{y}}})$ and $\Gamma(S,x,{{\mathbf{y}}},y)$ depend only on $k,d$ and $\ell=|{{\mathbf{y}}}|$ and, if e.g. $t(x)=0$ (i.e. $k$ is odd), $\Theta(S,x,{{\mathbf{y}}})=[{\mathfrak{m}}(k+1,\ell)]^{\ell}$ and $\Gamma(S,x,{{\mathbf{y}}},y)={\mathfrak{s}}(k+1,\ell)[{\mathfrak{m}}(k+1,\ell)]^{2\ell-2}+{[({\mathfrak{m}}(k+1,\ell))^{\ell}-{\mathfrak{m}}(k,d)]^{2}}$.

The last point uses that if ${\mathcal{C}}_{x}={{\mathbf{y}}}$ with $|{{\mathbf{y}}}|=\ell$, then $|{\mathcal{C}}_{f(y)}|=|{\mathcal{C}}_{x}|=\ell$ for all $y\in{{\mathbf{y}}}$. ∎

6.3. Symmetric minimax values

Here we discuss the formulas introduced in Subsection 2.14.

We fix some value $a\in(0,1)$. For $S\in{\mathcal{S}}_{f}$, we build the family $(m_{a}(S,x))_{x\in S}$ by induction, starting from the root, setting $m_{a}(S,r)=a$ and, for all $x\in S\setminus\{r\}$,

[TABLE]

Observe that $m_{a}(S,x)$ actually depends only on $K_{x}^{S}$, i.e. $m_{a}(S,x)=m_{a}(K^{S}_{x},x)$

Example 26**.**

Consider a possibly random tree ${\mathcal{T}}\in{\mathcal{S}}_{f}$ enjoying the property that for any $S\in{\mathcal{S}}_{f}$ with leaves $L_{S}$, the family $({\mathcal{T}}_{x})_{x\in L_{S}}$ is independent conditionally on $A_{S}=\{S\subset{\mathcal{T}},{\mathcal{D}}_{L_{S}}=\emptyset\}$ as soon as $\Pr(A_{S})>0$. Fix $a\in(0,1)$ and assume that conditionally on ${\mathcal{T}}$, the family $(R(y))_{y\in{\mathcal{L}}}$ is independent and $R(y)\sim$ Bernoulli$(m_{a}({\mathcal{T}},y))$ for all $y\in{\mathcal{L}}$. Then Assumption 5 is fulfilled, and for any $S\in S_{f}$ such that $\Pr(A_{S})>0$ and any $x\in L_{S}$, we have $m(S,x)=m_{a}(S,x)$. We are generally not able to compute $s(S,x)$.

Observe that this is a qualitative symmetry assumption, saying that knowing ${\mathcal{T}}$, for any $v\in{\mathcal{T}}\setminus{\mathcal{L}}$, the family of the minimax values $(R(x),x\in{\mathcal{C}}_{v})$ is i.i.d. Once this is assumed, the only remaining parameter is the mean minimax rating of the root (which we set to $a$).

Once the value $a=m_{a}({\mathcal{T}},r)$ is chosen (even if not knowing ${\mathcal{T}}$), it is easy to make the algorithm compute the necessary values of $m_{a}$, as explained in Subsection 2.14: each time a new node $x$ of ${\mathcal{T}}$ is created by the algorithm, we can compute $m_{a}({\mathcal{K}}_{x},x)$ from $m_{a}({\mathcal{K}}_{f(x)},f(x))$ and $|{\mathcal{C}}_{f(x)}|$.

Proof.

Assumption 5 is satisfied because (a) the random tree ${\mathcal{T}}$ is supposed to satisfy the required independence property and (b) conditionally on ${\mathcal{T}}$, for any $x\in{\mathcal{L}}$, $m_{a}({\mathcal{T}},x)$ depends only on ${\mathcal{K}}_{x}$.

It remains to verify that $m(S,x)=m_{a}(S,x)$ for all $S\in{\mathcal{S}}_{f}$ of which $x$ is a leave.

We first show by backward induction that $\Pr(R(x)=1|{\mathcal{T}})=m_{a}({\mathcal{T}},x)$ for all $x\in{\mathcal{T}}$. First, this is obvious if $x\in{\mathcal{L}}$ by construction. Next, if this is true for all the children (in ${\mathcal{T}}$) of $x\in{\mathcal{T}}\setminus{\mathcal{L}}$ with e.g. $t(x)=1$, then we have $\Pr(R(x)=1|{\mathcal{T}})=1-\prod_{y\in{\mathcal{C}}_{x}}\Pr(R(y)=0|{\mathcal{T}})=1-\prod_{y\in{\mathcal{C}}_{x}}(1-m_{a}({\mathcal{T}},y))$. We first used that the family $(R(y))_{y\in{\mathcal{C}}_{x}}$ is independent conditionally on ${\mathcal{T}}$ and then the induction assumption. Using finally (11) (recall that $t(x)=1$ and that $f(y)=x$ for all $y\in{\mathcal{C}}_{x}$), we find $\Pr(R(x)=1|{\mathcal{T}})=1-\prod_{y\in{\mathcal{C}}_{x}}(1-[1-(1-m_{a}({\mathcal{T}},x))^{1/|{\mathcal{C}}_{x}|}])=m_{a}({\mathcal{T}},x)$.

Fix now $S\in S_{f}$ such that $\Pr(A_{S})>0$, where $A_{S}=\{S\subset{\mathcal{T}},{\mathcal{D}}_{L_{S}}=\emptyset\}$ and $x\in L_{S}$. Since $A_{S}\in\sigma({\mathcal{T}})$, we deduce that $m(S,x)=\Pr(R(x)=1|A_{S})=\mathbb{E}[\Pr(R(x)=1|{\mathcal{T}})|A_{S}]=\mathbb{E}[m_{a}({\mathcal{T}},x)|A_{S}]$. But we know that $m_{a}({\mathcal{T}},x)=m_{a}({\mathcal{K}}_{x},x)$. Since ${\mathcal{K}}_{x}=K_{x}^{S}$ on $A_{S}$, we conclude that $m(S,x)=m_{a}(K_{x}^{S},x)=m_{a}(S,x)$ as desired. ∎

Let us mention that Pearl’s model, which we already interpreted as a particular case of Example 23, can also be seen as a particular case of Example 26, where we can furthermore compute $s$.

Remark 27**.**

Consider again Pearl’s model [20]: ${\mathcal{T}}$ is the deterministic regular tree with degree $d\geq 2$ and depth $K\geq 1$ and the family $(R(x))_{x\in{\mathcal{L}}}$ is i.i.d. Bernoulli$(p)$-distributed. Then we already know that for all $S\in{\mathcal{S}}_{f}$ and $x\in L_{S}$ such that $\Pr(A_{S})>0$, we have $m(S,x)={\mathfrak{m}}(|x|)$ and $s(S,x)={\mathfrak{s}}(|x|)$, with ${\mathfrak{m}}$ and ${\mathfrak{s}}$ as in Remark 24. One then also has, for all $S\in{\mathcal{S}}_{f}$ and $x\in L_{S}$ such that $\Pr(A_{S})>0$, if $x\neq r$, denoting by $v=f(x)$ and $S_{v}=S\setminus C^{S}_{v}$,

[TABLE]

Setting $a={\mathfrak{m}}(0)$, which can be computed from $p,K,d$, we thus have $m(S,x)=m_{a}(S,x)$ as defined in (11), and we can compute $s(S,x)$. Note that it is not necessary to determine precisely $s(\{r\},r)$: we can set $s(\{r\},r)=1$ (or any other positive constant) by Remark 14.

Indeed, the above formulas are nothing but a complicated version of the ones in Remark 24, since we have $m(S,x)={\mathfrak{m}}(|x|)$, $s(S,x)={\mathfrak{s}}(|x|)$, $|C^{S}_{v}|=d$, $m(S_{v},v)={\mathfrak{m}}(|v|)$ and $s(S_{v},v)={\mathfrak{s}}(|v|)$.

There are other cases where we can characterize $s$, which should thus be numerically computable.

Example 28**.**

Assume that ${\mathcal{T}}$ is a homogeneous Galton-Watson tree with reproduction law $\mu$ such that $\sum_{\ell\geq 1}\ell\mu(\ell)\leq 1$ and $\mu(0)>0$, so that ${\mathcal{T}}$ is a.s. finite. Fix $a\in(0,1)$ and assume that conditionally on ${\mathcal{T}}$, the family $(R(y))_{y\in{\mathcal{L}}}$ is independent and that $R(y)\sim$ Bernoulli$(m_{a}({\mathcal{T}},y))$ for all $y\in{\mathcal{L}}$. Then for all $S\in S_{f}$ such that $\Pr(A_{S})>0$ and all $x\in L_{S}$, we have $m(S,x)=m_{a}(S,x)$ and $s(S,x)={\mathfrak{s}}(1-m_{a}(S,x)){{\mathbf{1}}}_{\{t(x)=0\}}+{\mathfrak{s}}(m_{a}(S,x)){{\mathbf{1}}}_{\{t(x)=1\}}$, where ${\mathfrak{s}}$ is the unique function from $[0,1]$ into $[0,1]$ such that for all $\alpha\in[0,1]$,

[TABLE]

Proof.

We already know from Example 26 that Assumption 5 is satisfied and that $m(S,x)=m_{a}(S,x)$. Next, (12) has a unique solution because the map $F:E\mapsto E$, where $E$ is the set of all functions from $[0,1]$ into $[0,1]$, defined by

[TABLE]

is a contraction. Indeed, $||F({\mathfrak{s}}_{1})-F({\mathfrak{s}}_{2})||_{\infty}\leq\kappa||{\mathfrak{s}}_{1}-{\mathfrak{s}}_{2}||_{\infty}$ with $\kappa=\sum_{\ell\geq 1}\mu(\ell)<1$.

Let us denote by ${\mathfrak{s}}(a)=s(\{r\},r)$, which clearly depends only on $a$ (and $\mu$).

For any $S\in S_{f}$ such that $\Pr(A_{S})>0$ and $x\in L_{S}$, we have $s(S,x)={\mathfrak{s}}(m_{a}(S,x))$ if $t(x)=1$ and $s(S,x)={\mathfrak{s}}(1-m_{a}(S,x))$ if $t(x)=0$. Indeed, the law of $({\mathcal{T}}_{x},(R(u))_{u\in{\mathcal{L}}\cap{\mathcal{T}}_{x}})$ conditionally on $A_{S}$ is the same as that of $({\mathcal{T}},(R(u))_{u\in{\mathcal{L}}})$ (re-rooted at $x$), replacing $a$ by $m_{a}(S,x)$: ${\mathcal{T}}_{x}$ is a Galton-Watson tree with reproduction law $\mu$ and, knowing $A_{S}$ and ${\mathcal{T}}_{x}$, one easily checks that $m_{a}({\mathcal{T}},y)=m_{m_{a}(S,x)}({\mathcal{T}}_{x},y)$ for all $y\in{\mathcal{L}}\cap{\mathcal{T}}_{x}$. Hence we have $s(S,x)={\mathfrak{s}}(m_{a}(S,x))$ if $t(x)=1$. If now $t(x)=0$, we see that $s(S,x)={\mathfrak{s}}(1-m_{a}(S,x))$ by exchanging the roles of the two players.

It remains to verify that ${\mathfrak{s}}$ satisfies (12). To this end, it suffices to apply the formula of Lemma 18 concerning $s$ with $S=\{r\}$ (whence $A_{S}=\Omega$) and $x=r$ (with $t(r)=1$) and to observe that

$\bullet$ $s(\{r\},r)={\mathfrak{s}}(a)$ and $m(\{r\},r)=a$,

$\bullet$ $\Pr(r\in{\mathcal{L}},R(r)=1)=\mu(0)a$ and $\Pr(r\in{\mathcal{L}},R(r)=0)=\mu(0)(1-a)$,

$\bullet$ $\sum_{{{\mathbf{y}}}\subset{\mathbb{C}}_{r},|{{\mathbf{y}}}|=\ell}\Pr({\mathcal{C}}_{r}={{\mathbf{y}}})=\mu(\ell)$,

$\bullet$ for any ${{\mathbf{y}}}\subset{\mathbb{C}}_{r}$ with $|{{\mathbf{y}}}|=\ell\geq 1$, conditionally on ${\mathcal{C}}_{r}={{\mathbf{y}}}$, for all $y\in{{\mathbf{y}}}$, we have $m_{a}(\{r\}\cup{{\mathbf{y}}},y)=1-(1-a)^{1/\ell}$ and thus, since $t(y)=1$, $s(\{r\}\cup{{\mathbf{y}}},y)={\mathfrak{s}}((1-a)^{1/\ell})$, whence $\Gamma(\{r\},r,{{\mathbf{y}}},y)={\mathfrak{s}}((1-a)^{1/\ell})(1-a)^{2(\ell-1)/\ell}+[(1-a)^{\ell/\ell}-(1-a)]^{2}=(1-a)^{2(\ell-1)/\ell}{\mathfrak{s}}((1-a)^{1/\ell})$. ∎

It does not seem easy to solve (12). However, here is one possibility.

Remark 29**.**

Assume that ${\mathcal{T}}$ is a homogeneous Galton-Watson tree with reproduction law $\mu=(1-p)\delta_{0}+p\delta_{d}$, with $d\geq 2$ and $p\in(0,1/d]$. Consider the unique solution $a_{0}\in(0,1)$ to $a_{0}=(1-a_{0})^{1/d}$. Assume that conditionally on ${\mathcal{T}}$, the family $(R(y))_{y\in{\mathcal{L}}}$ is independent and that $R(y)\sim$ Bernoulli$(m_{a_{0}}({\mathcal{T}},y))$ for all $y\in{\mathcal{L}}$. Then for all $S\in S_{f}$ such that $\Pr(A_{S})>0$ and all $x\in L_{S}$, we have $m(S,x)=m_{a_{0}}(S,x)$ and $s(S,x)={\mathfrak{s}}(a_{0})=(1-p)a_{0}^{d+1}/[1-pa_{0}^{2d-2}]$ is constant.

Indeed, in such a case (12) rewrites as

[TABLE]

whence ${\mathfrak{s}}(a_{0})=(1-p)a_{0}^{d+1}/[1-pa_{0}^{2d-2}]$. Also, one easily checks that for any $S\in{\mathcal{S}}_{f}$ such that $\Pr(A_{S})>0$ (so that $S$ is $d$-regular) and any $x\in L_{S}$, we have $m_{a_{0}}(S,x)=a_{0}$ if $t(x)=1$ and $m_{a_{0}}(S,x)=1-a_{0}$ if $t(x)=0$. This of course uses (11) and that $1-(1-a_{0})^{1/d}=1-a_{0}$ and $(1-a_{0})^{1/d}=a_{0}$. Consequently, $s(S,x)$ always equals ${\mathfrak{s}}(a_{0})$: $s(S,x)={\mathfrak{s}}(m_{a_{0}}(S,x))={\mathfrak{s}}(a_{0})$ if $t(x)=1$ and $s(S,x)={\mathfrak{s}}(1-m_{a_{0}}(S,x))={\mathfrak{s}}(a_{0})$ if $t(x)=0$.

7. Global optimality fails

Proof of Remark 17.

We assume here that ${\mathcal{T}}$ is the binary tree with depth $3$. We thus have the eight leaves $111$, $112$, $121$, $122$, $211$, $212$, $221$, $222$ (recall Subsection 2.1). We also assume that the family $(R(x))_{x\in{\mathcal{L}}}$ is i.i.d. with common law Bernoulli$(1/2)$.

Observe that ${\mathcal{T}}$ can be seen as an inhomogeneous Galton-Watson tree with reproduction laws $\mu_{0}=\mu_{1}=\mu_{2}=\delta_{2}$ and $\mu_{3}=\delta_{0}$. Applying Example 23 (with $q_{3}=1/2$, the values of $q_{0},q_{1},q_{2}$ being irrelevant), we can compute the functions $m$ and $s$: for any $S\in{\mathcal{S}}_{f}$ such that $\Pr(A_{S})>0$ and $x\in L_{S}$, we have $m(S,x)={\mathfrak{m}}(|x|)$ and $s(S,x)={\mathfrak{s}}(|x|)$, where

[TABLE]

The value of ${\mathfrak{m}}(0)$ and ${\mathfrak{s}}(0)$ are not useful to the algorithm. Let us however notice that $\mathbb{E}[R(r)]={\mathfrak{m}}(0)=207/256$.

By symmetry, we can replace the uniformly random matches (starting from some $z$) used in any admissible algorithm, see Definition 3, by the visit of any deterministic leave (under $z$), without changing (at all) the performance of the algorithm. With this slight modification, some tedious computations show that Algorithm 13, using the above function $m$ and $s$, leads to the following strategy (and results) for the three first steps.

Visit the leave $x_{1}=111$.

If $R(x_{1})=1$ (whence $R_{1}(r)=228/256$), then

$\{$ visit the leave $x_{2}=121$.

If $R(x_{2})=1$ (whence $R_{2}(r)=1$), then

$\{$ stop here (or visit any other leave). We have $R_{3}(r)=1.$ $\}$

If $R(x_{2})=0$ (whence $R_{2}(r)=200/256$), then

$\{$ visit the leave $x_{3}=122$. ​​We have $R_{3}(r)=1$ if $R(x_{3})=1$ and $R_{3}(r)=144/256$

if $R(x_{3})=0$. $\}$ $\}$

If $R(x_{1})=0$ (whence $R_{1}(r)=186/256$), then

$\{$ visit the leave $x_{2}=112$.

If $R(x_{2})=1$ (whence $R_{2}(r)=228/256$), then

$\{$ visit the leave $121$. We have $R_{3}(r)=1$ if $R(x_{3})=1$ and $R_{3}(r)=200/256$

if $R(x_{3})=0$. $\}$

If $R(x_{2})=0$ (whence $R_{2}(r)=144/256$), then

$\{$ visit the leave $x_{3}=211$. We have $R_{3}(r)=192/256$ if $R(x_{3})=1$

and $R_{3}(r)=96/256$ if $R(x_{3})=0$. $\}$ $\}$

Noting that $\mathbb{E}[(R_{n}(r)-R(r))^{2}]=\mathbb{E}[(R(r))^{2}]-\mathbb{E}[(R_{n}(r))^{2}]=207/256-\mathbb{E}[(R_{n}(r))^{2}]$ because $\mathbb{E}[R(r)R_{n}(r)]=\mathbb{E}[(R_{n}(r))^{2}]$ (since $R_{n}(r)=\mathbb{E}[R(r)|{\mathcal{F}}_{n}]$) and since $\mathbb{E}[(R(r))^{2}]=\mathbb{E}[R(r)]={\mathfrak{m}}(0)=207/256$, we conclude that

[TABLE]

The following strategy, that we found with the help of a computer, is less efficient in two steps but more efficient in three steps. We use the notation ${\tilde{R}}_{n}(r)$ as in the statement.

Visit the leave ${\tilde{x}}_{1}=111$.

If $R({\tilde{x}}_{1})=1$ (whence ${\tilde{R}}_{1}(r)=228/256$), then

$\{$ visit the leave ${\tilde{x}}_{2}=121$.

If $R({\tilde{x}}_{2})=1$ (whence ${\tilde{R}}_{2}(r)=1$), then

$\{$ stop here (or visit any other leave). We have ${\tilde{R}}_{3}(r)=1.$ $\}$

If $R({\tilde{x}}_{2})=0$ (whence ${\tilde{R}}_{2}(r)=200/256$), then

$\{$ visit the leave ${\tilde{x}}_{3}=122$. ​​We have ${\tilde{R}}_{3}(r)=1$ if $R({\tilde{x}}_{3})=1$ and ${\tilde{R}}_{3}(r)=144/256$

if $R({\tilde{x}}_{3})=0$. $\}$ $\}$

If $R({\tilde{x}}_{1})=0$ (whence ${\tilde{R}}_{1}(r)=186/256$), then

$\{$ visit the leave ${\tilde{x}}_{2}=211$.

If $R({\tilde{x}}_{2})=1$ (whence ${\tilde{R}}_{2}(r)=216/256$), then

$\{$ visit the leave $221$. We have ${\tilde{R}}_{3}(r)=1$ if $R({\tilde{x}}_{3})=1$ and ${\tilde{R}}_{3}(r)=176/256$

if $R({\tilde{x}}_{3})=0$. $\}$

If ${\tilde{R}}({\tilde{x}}_{2})=0$ (whence ${\tilde{R}}_{2}(r)=156/256$), then

$\{$ visit the leave ${\tilde{x}}_{3}=112$. We have ${\tilde{R}}_{3}(r)=216/256$ if $R({\tilde{x}}_{3})=1$

and ${\tilde{R}}_{3}(r)=96/256$ if $R({\tilde{x}}_{3})=0$. $\}$ $\}$

We conclude, using the same argument as previously, that

[TABLE]

The proof is complete. ∎

8. Numerical results

8.1. Numerical problems

Algorithm 13 is subjected to numerical problems due to the fact that it proceeds to a high number of multiplications of reals in $[0,1]$. For example, the algorithm continuously computes products of the form $\prod_{k=1}^{d}r_{k}$ and $1-\prod_{k=1}^{d}(1-r_{k})$, with $r_{1},\dots,r_{d}\in[0,1]$. If coded naively, it immediately finds [math] or $1$ and does not work at all. We overcame such problems with the change of variables $\phi(r)=\log[r/(1-r)]$. Everywhere, we used $\phi(R)$, $\phi(U)$ and $\phi(Z)$ instead of $R$, $U$ and $Z$ (we mean, concerning the values $R_{n}(x)$, $U_{n}(x)$ and the $Z_{n}(x)$). Actually, for large games, some numerical problems persist: at some steps, we have numerically $(R_{n+1}(r),U_{n+1}(r),Z_{n+1}(r))=(R_{n}(r),U_{n}(r),Z_{n}(r))$ (even after the change of variables), which should never be the case. However, the above trick eliminates most of them. Instead of using $\phi$, one could manipulate simultaneously $\log r$ and $\log(1-r)$. This would be more or less equivalent, the use of $\phi$ is just slightly more concise.

We used the following expressions. We carefully separated different cases, because e.g. for $u$ very large (say, $u\geq 750$), the computer answers $\log(1+\exp(u))=+\infty$ but $u+\log(1+\exp(-u))=u$, these two quantities being theoretically equal. Consider $r,s\in[0,1]$ and $u=\phi(r)$, $v=\phi(s)$. We e.g. assume that $0\leq s\leq r\leq 1$ (whence $-\infty\leq v\leq u\leq+\infty$) and we naturally allow $\phi(0)=-\infty$, $\phi(1)=+\infty$, $\exp(-\infty)=0$, etc. Observe that $r+s\leq 1$ if and only if $u+v\leq 0$.

(a) $\phi(1-r)=-\phi(r)$,

(b) \phi(rs)=\begin{cases}u+v-\log(1+e^{u}+e^{v})&\hbox{if u<0,}\\ +\infty&\hbox{if v=+\infty,}\\ v-\log(1+e^{-u}+e^{v-u})&\hbox{if v<+\infty$and$u\geq 0}.\end{cases}

(c) \phi(r+s)=\begin{cases}-\infty&\hbox{if u=-\infty},\\ +\infty&\hbox{if v=-\infty$and$u=+\infty},\\ u+\log(1+e^{v-u}+2e^{v})-\log(1-e^{u+v})&\hbox{if u>-\infty$and$u+v\leq 0}.\end{cases}

(d) \phi(s/r)=\begin{cases}+\infty&\hbox{if u=v},\\ v&\hbox{if v<u=+\infty},\\ \log(1+e^{u})-u+v-\log(1-e^{v-u})&\hbox{if v<u<0},\\ \log(1+e^{-u})+v-\log(1-e^{v-u})&\hbox{if 0\leq u<+\infty$and$v<u}.\end{cases}

(e) \log r=\begin{cases}u-\log(1+e^{u})&\hbox{if u<0,}\\ -\log(1+e^{-u})&\hbox{if u\geq 0.}\end{cases}

We can compute $\phi(r-s)=\phi(r(1-s/r))$ from $u,v$ using (a), (b) and (d). For $r_{1},\dots,r_{d}\in[0,1]$, we can compute $\phi(\prod_{1}^{d}r_{k})$ from the values of $u_{k}=\phi(r_{k})$ recursively using (b), and we can get $\phi(1-\prod_{1}^{d}(1-r_{k}))=-\phi(\prod_{1}^{d}(1-r_{k}))$ using furthermore (a). Etc.

8.2. The algorithms

We will make play some versions of our algorithm against the two versions of MCTS recalled in Section 9 on a few real games (variations of Connect Four) and on Pearl’s model.

We call MCTS($a,b$) Algorithm 31 and MCTS’($a,b$) Algorithm 30 with the parameters $a,b>0$.

We call GW (resp. GW2) Algorithm 13 with the functions $m$ and $s$ of Example 23 (resp. Example 25), assuming that the game is the realization of an inhomogeneous Galton-Watson tree (resp. inhomogeneous Galton-Watson tree of order $2$). The functions ${\mathfrak{m}}$ and ${\mathfrak{s}}$ are computed as follows, for example in the case of GW. We handle a large number ($10^{8}$) of uniformly random matches starting from the initial configuration of the game, this allows us to estimate $\mu_{0},\dots,\mu_{K}$. The values $q_{0},\dots,q_{K}$ are trivial for Connect Four, since we simply have $q_{k}={{\mathbf{1}}}_{\{k\hbox{ \tiny is odd}\}}$. We then use the formulas stated in Example 23. We obtain rather stable results. Of course, this is done once for all for each game.

We call Sym Algorithm 13 with the function $m=m_{a}$ defined by (11), with the choice $a=1/2$, and with $s\equiv 1$. Recall that $a\in(0,1)$ is the expected minimax rating of the root. This is the most simple and universal algorithm, although not fully theoretically justified (see however Remark (29) in Subsection 6.3).

We finally call SymP Algorithm 13 with the functions $m$ and $s$ defined in Remark 27, here again with $a=1/2$. This is the theoretical algorithm furnished by our study in the case of Pearl’s game (if $a=1/2$) and it is precisely the same as GW in this case, see Remarks 24 and 27.

Let us now give a few precisions.

(i) In all the experiments below, each algorithm keeps the information provided by its own simulations handled to decide its previous moves. In practice, this at most doubles the quantity of information (when compared to the case where we would delete everything at each new move), because most of the previous simulations led to other positions.

(ii) Concerning Sym and SymP, we actually use $a=1/2$ as expected minimax value of the true root of the game, that is the true initial position. When in another configuration $x$, we use $m_{a}({\mathcal{T}},x)$ (with $a=1/2$, here ${\mathcal{T}}$ is the tree representing the whole game) as expected minimax value of the current root $x$ (i.e. the current position of the game). Such a value is automatically computed when playing the game.

8.3. The numerical experiments

First, let us mention that we handled many trials using GW and GW2. They almost always worked less well than Sym concerning Connect Four, so we decided not to present those results. Also, concerning Sym and SymP, we tried other values for the expected minimax value $a\in(0,1)$ of the root without observing significantly better results, so we always use $a=1/2$. Similarly, we experimented other values for $b\in{\mathbb{R}}$ (see Subsection 2.14, Sym corresponds to the case $b=0$ and SymP to the case where $b=2$), here again without clear success.

In each subsection below (except Subsection 8.11), which concerns one given game, we proceed as follows.

For each given amount of time per move, we first fit the parameters of MCTS. To this aim, we perform a championship involving MCTS($a,b$), for all $a=k/2$, $b=\ell/2$ with $1\leq k\leq\ell\leq 10$ (we thus have $55$ players). Each player competes $40$ times against all the other ones ($20$ times as first player, $20$ times as second one), and we select the player with the highest number of victories. Observe that each player participates to $2160$ matches. The resulting best player is rather unstable, but we believe this is due to the fact that the competition is very tight among a few players. Hence even if we do not select the true best player, we clearly select a very good one. Note that we impose $a\leq b$ because, after many trials allowing $a>b$, the best player was always of this shape.

Of course, we do exactly the same thing to fit the parameters of MCTS’.

Then, we make our algorithm (Sym or SymP) compete against the best MCTS and the best MCTS’, $10000$ times as first player and $10000$ times as second one.

Also, we indicate the (rounded) mean number of iterations made by each algorithm at the first move. This sometimes looks ridiculous: when e.g. playing a version of Connect Four with large degree with $1$ millisecond per move, this mean number of iterations is $8$ for Sym and $11$ for MCTS. However, after sufficiently many moves, this mean number of iterations becomes much higher. In other words, the algorithms more or less play at random at the beginning, but become more and more clever as the game progresses. So in some sense, the algorithm that wins is the one becoming clever before the other.

Finally, let us explain how to read the tables below, which are all of the same shape. For example, the first table, when playing a large Pearl game, is as follows.

[TABLE]

Each player had $1$ millisecond to decide each of its moves. The championships were won by MCTS(3.5,5) and MCTS’(2,3). When playing its first move, MCTS(3.5,5) (resp. MCTS’(2,3), resp. SymP) proceeded in mean to 498 (resp. 442, resp. 163) iterations.

When playing first, SymP won 5882 times and lose 4118 times against MCTS(3.5,5), and there has been 0 draw. When playing first, MCTS(3.5,5) won 4340 times and lose 5660 times against SymP, and there has been 0 draw.

When playing first, SymP won 4992 times and lose 5008 times against MCTS’(2,3), and there has been 0 draw. When playing first, MCTS’(2,3) won 5221 times and lose 4779 times against SymP, and there has been 0 draw.

Finally, the results in bold mean that our algorithm (Sym or SymP) is beaten.

8.4. Large Pearl game

We first consider Pearl’s game, that is the game of Example 24 with the regular tree with degree $d=2$ and depth $K=32$, with i.i.d. Bernoulli$(p)$ random variables on the leaves and with $p$ such that $\mathbb{E}[R(r)]=1/2$ ($p$ is easily computed numerically using the formula of Remark 24 and that $p={\mathfrak{m}}(K)$ and $1/2={\mathfrak{m}}(0)$).

Here and in the next subsection, to be as fair as possible, in each cell of the tables below, each match is played in both senses on a given realization of the model, so that if the two opponents were playing perfectly, one would find twice the same results in each cell.

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

We observe that SymP is better than MCTS but beats MCTS’ only when the amount of time per play is high enough.

For this game, SymP performs around 4 times fewer iterations per unit of time than MCTS.

8.5. Small Pearl game

We next consider a much smaller Pearl game: as previously, $d=2$ and $p$ is chosen is such that $\mathbb{E}[R(r)]=1/2$, but the depth of the tree is $K=16$ (instead of $K=32$).

Here, and only here, due to the smallness of the game, we needed to modify the way we fit the parameters of MCTS and MCTS’. Namely, we perform a championship involving MCTS($a,b$), for all $a=k/2$, $b=\ell/2$ with $1\leq k\leq\ell\leq 20$ (we thus have $210$ players). Each player competes $20$ times against all the other ones ($10$ times as first player, $10$ times as second one), and we select the player with the highest number of victories. Each player participates to $4180$ matches.

Proceeding as in the previous subsection, we found the following results.

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

Here it seems that SymP almost always finds the winning strategy at the first move with $16$ ms, this explains why the number of iterations is so small (at $16$ and $32$ ms): SymP stops before $16$ ms are elapsed. We thus believe it always finds it with $32$ ms.

At $32$ ms, it seems that MCTS’ also always found the winning strategy among $5083$ times it started the game with a possible winning strategy. MCTS missed $24$ times the winning strategy among $5034$ times it started the game with a possible winning strategy.

Observe that SymP is better than MCTS but, here again, beats MCTS’ only when the amount of time per play is high enough.

Also, even if the game is theoretically fair (because $\mathbb{E}[R(r)]=1/2$), it seems easier, for all the algorithms, to find the winning strategy when being the second player (this can be observed until $8$ ms). This might be explained by the fact that the second player is the one playing the last move.

Here also, SymP performs around $4$ times fewer iterations per unit of time than MCTS.

8.6. Standard Connect Four

We now play Connect Four in its usual version: we have $7$ columns, $6$ lines, and the goal is to connect (horizontally, vertically, or diagonally) $4$ discs.

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

We are largely beaten, and this is worse and worse as the given amount of time increases. Observe however the high number of draws, which indicates that even if MCTS almost always wins at the end, the competition is tight.

Let us mention that with $1024$ ms per move, we found, for Sym vs MCTS(10,10): 14/986(0), 925/74(1) and, for Sym vs MCTS’(10,10): 14/986(0), 924/74(2): we are destroyed.

Sym performs here around twice fewer iterations per unit of time than MCTS.

8.7. A version of Connect Four with small degree

We next consider the variation of Connect Four with $4$ columns, $10$ lines, and where the goal is to connect (horizontally, vertically, or diagonally) $3$ discs.

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

Here we observe that MCTS and MCTS’ win when having a lot time (for such a small game, $1$ millisecond is much), but when the amount of time is so high that we are close to finding the winning strategy at the first move, Sym is better. This might be due to the fact that Sym automatically does some pruning.

Here Sym performs around $3$ or $4$ times fewer iterations per unit of time than MCTS. This is not the case with $16$ ms because it finds the right move, and thus stops, before the 16 ms are elapsed.

8.8. A first version of connect 4 with large degree

Here we consider a very simple version of Connect Four, with $15$ columns, $15$ lines, and where the goal is to connect (horizontally, vertically, or diagonally) $3$ discs. A human player immediately finds the winning strategy when playing first. However, for a computer, the situation is not so easy, because there are many possibilities (if not taking advantage of the symmetries of the game).

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

Here we are really better than MCTS. We believe this is due to the fact that the degree of the game is very large.

For this game, Sym performs around $2$ times fewer iterations per unit of time than MCTS.

8.9. A second version of Connect Four with large degree

We consider here the version of Connect Four with $15$ columns, $6$ lines, and where the goal is to connect (horizontally, vertically, or diagonally) $5$ discs. Here the situation is intractable for a normal human player.

[TABLE]

[TABLE]

[TABLE]

[TABLE]

[TABLE]

The number of iterations seems very small here, but let us recall that this concerns only the beginning of the game. As already mentioned, this number of iterations actually increases considerably when approaching the end of the game.

We observe that Sym is really better than MCTS and MCTS’ when the amount of time per move is small. When this amount of time increases, we are beaten. We have two main explanations for this, see Subsection 8.12.

Here Sym performs around twice fewer iterations per unit of time than MCTS.

8.10. Inverse and large Connect Four

We next consider the inverse version of Connect Four with $15$ columns, $6$ lines, and where the first player that connects (horizontally, vertically, or diagonally) $5$ discs looses. Although this modification is coded very easily (only one line has to be modified so that win and loss are exchanged), this considerably modifies the game. The algorithms (MCTS, MCTS’ and Sym) all take this into account immediately: at the beginning of a match, one usually sees the algorithms play in the middle of the board, while they play near the extremities in the inverse case.

[TABLE]

[TABLE]

[TABLE]

Here again, we observe that Sym wins when the amount of time per move is very small.

8.11. Back to Pearl’s game: rate of convergence

Here we consider Pearl’s game with $d=2$, $K$ even and with the value $p=(\sqrt{5}-1)/2\simeq 0.618$. With this particular value of $p$, Pearl [20] showed that $\mathbb{E}[R(r)]=p$ and that the expected required number of visited leaves for AlphaBeta to determine $R(r)$ equals $(2/(\sqrt{5}-1))^{K}$. Here $r$ is the true root of the game. In the whole subsection, we use SymP with the correct value of $a$, i.e. $a=p$.

First, we call $\tau_{K}$ the number of leaves that SymP needs to determine $R(r)$. Denoting by $\bar{\tau}_{K}$ the average value over $10000$ trials, we found

[TABLE]

I thus seems highly plausible that our algorithm visits the leaves in the same order as AlphaBeta (for a Pearl game), up to some random permutation. This is rather satisfying, since Tarsi [23] showed that for a Pearl game, AlphaBeta is optimal in the sense of the expected number of leaves necessary to determine $R(r)$.

Finally, we plot a Monte-Carlo approximation (with $10000$ trials) of $\mathbb{E}[(R_{n}(r)-R(r))^{2}]$ as a function of the number $n$ of iterations, when $K=8$ and $K=16$, as well as one trajectory of $n\mapsto R_{n}(r)$ when $K=16$.

$K=8$                                                $K=16$ $\mathbb{E}[(R_{n}(r)-R(r))^{2}]$ as a function of the number $n$ of iterations.

$K=16$ One trajectory of $n\mapsto R_{n}(r)$.

8.12. Conclusion

When playing Pearl’s game, SymP beats MCTS and seems competitive against MCTS’. This is reassuring, since our algorithms are typically designed for such games.

On true games, MCTS and MCTS’ seem globally much better than Sym. However, we found two situations where Sym may win.

The first and most interesting situation is the one where the game is so large (or the amount of time so small) that very few iterations can be performed by the challengers. This is quite natural, and there are two possible reasons for that.

$\bullet$ Our algorithm is only optimal step by step. So, it is absolutely not clear that it works well when we have enough time to handle many iterations.

$\bullet$ Assumption 5 imposes some independence properties. While this is reasonable, on any game, in some sense to be precised, after a small number of iterations (when playing a small number of uniformly random matches, it is rather clear that the issues will be almost independent), this is clearly not the case when performing a large number of well-chosen matches.

The second situation is that where the game is so small that we can hope to find the winning strategy at the first move, and where Sym may find it before MCTS and MCTS’. As already mentioned, we believe this is due to the fact that Sym does some pruning.

From a theoretical point of view, it would be very interesting to study more relevant models such as the one proposed by Devroye-Kamoun [12]. Clearly, this falls completely out of our scope. On the contrary, it does not seem completely desesperate to find empirically variants of our algorithm that work much better in practise. For example, it may be relevant to use other choices of functions $m$ and $s$, to try a clever default policy, etc.

9. Appendix: Monte Carlo Tree Search algorithms

In this subsection, we write down precisely the versions of the MCTS algorithm we used to test our algorithm. We start with a modified version, more close to our study, where we do not throw down any information.

Algorithm 30** (Modified MCTS).**

Consider $\phi:{\mathbb{N}}\times{\mathbb{N}}\mapsto{\mathbb{R}}$, e.g. $\phi(w,c)=\frac{w+a}{c+b}$ for some $a,b>0$.

Step 1.* Simulate a uniformly random match from $r$, call $x_{1}$ the resulting leave and put ${{\mathbf{x}}}_{1}=\{x_{1}\}$.*

During this random match, keep track of $R(x_{1})$, of $B_{{{\mathbf{x}}}_{1}}=B_{rx_{1}}$ and of ${\mathcal{D}}_{{{\mathbf{x}}}_{1}}=\cup_{y\in B_{rx_{1}}}{\mathcal{H}}_{y}$ and set $C_{1}(x)=W_{1}(x)=0$ for all $x\in{\mathcal{D}}_{{{\mathbf{x}}}_{1}}$.

For all $x\in B_{x_{1}}$, set $C_{1}(x)=1$, $W_{1}(x)=R(x_{1})$.

Step n+1.* Put $z=r$ and do $z=\arg\!\max\{\phi(V_{n}(y),C_{n}(y)):y\in{\mathcal{C}}_{z}\}$ until $z\in{{\mathbf{x}}}_{n}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$, where*

[TABLE]

Set $z_{n}=z$.

(i) If $z_{n}\in{{\mathbf{x}}}_{n}$ (this will almost never occur if $n$ is reasonable for a large game), set ${{\mathbf{x}}}_{n+1}={{\mathbf{x}}}_{n}$, $B_{{{\mathbf{x}}}_{n+1}}=B_{{{\mathbf{x}}}_{n}}$ and ${\mathcal{D}}_{{{\mathbf{x}}}_{n+1}}={\mathcal{D}}_{{{\mathbf{x}}}_{n}}$

For all $x\in B_{rz_{n}}$, set $C_{n+1}(x)=C_{n}(x)+1$, $W_{n+1}(x)=W_{n}(x)+R({z_{n}})$.

For all $x\in(B_{{{\mathbf{x}}}_{n+1}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n+1}})\setminus B_{rz_{n}}$, set $C_{n+1}(x)=C_{n}(x)$, $W_{n+1}(x)=W_{n}(x)$.

(ii) Else (then $z_{n}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$), simulate a uniformly random match from $z_{n}$, call $x_{n+1}$ the resulting leave, set ${{\mathbf{x}}}_{n+1}={{\mathbf{x}}}_{n}\cup\{x_{n+1}\}$.

During this random match, keep track of $R(x_{n+1})$, of $B_{{{\mathbf{x}}}_{n+1}}=B_{{{\mathbf{x}}}_{n}}\cup B_{rx_{n+1}}$ and of ${\mathcal{D}}_{{{\mathbf{x}}}_{n+1}}=({\mathcal{D}}_{{{\mathbf{x}}}_{n}}\setminus\{z_{n}\})\cup\bigcup_{y\in B_{z_{n}x_{n+1}}\setminus\{z_{n}\}}{\mathcal{H}}_{y}$ and set $R_{n+1}(x)=W_{n+1}(x)=0$ for all $x\in\bigcup_{y\in B_{z_{n}x_{n+1}}\setminus\{z_{n}\}}{\mathcal{H}}_{y}$.

For all $x\in B_{rx_{n+1}}$, set $C_{n+1}(x)=C_{n}(x)+1$, $W_{n+1}(y)=W_{n}(x)+R(x_{n+1})$.

Finally, set $C_{n+1}(x)=C_{n}(x)$, $W_{n+1}(x)=W_{n}(x)$ for all $x\in(B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}})\setminus B_{rx_{n+1}}$.

Conclusion.* Stop after a given number of iterations $n_{0}$ (or after a given amount of time). As best child of $r$, choose $x_{*}=\arg\!\max\{\phi(V_{n_{0}}(x),C_{n_{0}}(x)):x$ child of $r\}$.*

Algorithm 30 updates the information on the whole visited branch at each new rollout. Here is a more standard version: it only creates, after each new simulation, one new node (together with its brothers) and updates the information only on the branch from the root to this new node. It seems clear that Algorithm 30 should be better, but it may lead to memory problems if the game is very large. We do not discuss such memory problems in the present paper.

Algorithm 31** (MCTS).**

Consider $\phi:{\mathbb{N}}\times{\mathbb{N}}\mapsto{\mathbb{R}}$, e.g. $\phi(w,c)=\frac{w+a}{c+b}$ for some $a,b>0$.

Step 1.* Simulate a uniformly random match from $r$, call $u$ the resulting leave.*

During this random match, keep track of $R(u)$ and of $T_{1}=\{r\}\cup{\mathcal{C}}_{r}$.

Set $C_{1}(x)=W_{1}(x)=0$ for all $x\in T_{1}\setminus B_{ru}$.

Set $C_{1}(x)=1$ and $W_{1}(x)=R(u)$ for all $x\in T_{1}\cap B_{ru}$.

Step n+1.* Put $z=r$ and do $z=\arg\!\max\{\phi(V_{n}(y),C_{n}(y)):y\in{\mathcal{C}}_{z}\}$ until $z\in L_{T_{n}}$, where*

[TABLE]

Set $z_{n}=z$.

(i) If $z_{n}\in{\mathcal{L}}$ (this will never occur if $n$ is reasonable for a large game), set $T_{n+1}=T_{n}$.

For all $x\in B_{rz_{n}}$, set $C_{n+1}(x)=C_{n}(x)+1$, $W_{n+1}(x)=W_{n}(x)+R(z_{n})$.

For all $x\in T_{n+1}\setminus B_{rz_{n}}$, set $C_{n+1}(x)=C_{n}(x)$, $W_{n+1}(x)=W_{n}(x)$.

(ii) If $z_{n}\notin{\mathcal{L}}$, simulate a uniformly random match from $z_{n}$, call $u$ the resulting leave, define $y$ as the child of $z_{n}$ belonging to $B_{z_{n}u}$ and set $T_{n+1}=T_{n}\cup{\mathcal{C}}_{z_{n}}$.

For all $x\in B_{rz_{n}}$, set $C_{n+1}(x)=C_{n}(x)+1$, $W_{n+1}(y)=W_{n}(x)+R(u)$.

Set $C_{n+1}(y)=1$, $W_{n+1}(y)={{\mathbf{1}}}_{\{R(u)=1\}}$ and, for all $x\in{\mathcal{H}}_{y}$, set $C_{n+1}(x)=W_{n+1}(x)=0$.

For all $x\in T_{n}\setminus B_{rz_{n}}$, set $C_{n+1}(x)=C_{n}(x)$ and $W_{n+1}(x)=W_{n}(x)$.

Conclusion.* Stop after a given number of iterations $n_{0}$ (or after a given amount of time). As best child of $r$, choose $x_{*}=\arg\!\max\{\phi(V_{n_{0}}(x),C_{n_{0}}(x)):x$ child of $r\}$.*

Of course, in both algorithms, the choice of the function $\phi$ is debatable. The choice $\phi(w,c)=(w+a)/(c+b)$, with $a>0$ and $b>0$ chosen empirically, seems to be a very good choice and was proposed by Lee et al [18, Subsection II-B-1].

Algorithm 30 is not admissible in the sense of Definition 3 because it may take different decisions with the same information. Indeed, it might visit twice the same leave consecutively: this does not modify the information but changes the values of $W_{n}$ and $C_{n}$. However, it is almost admissible: it would suffice to forbid two visits at the same leave (or alternatively to set $C_{n+1}(x)=C_{n}(x)$ and $W_{n+1}(x)=W_{n}(x)$ for all $x\in B_{{{\mathbf{x}}}_{n}}\cup{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$ in the case where $x_{n+1}\in{{\mathbf{x}}}_{n}$) to make it admissible. Since such a double visit almost never happens in practice, we decided not to complicate the definition of admissible algorithms nor to modify Algorithm 30.

Algorithm 31 is not admissible, because it has not the good structure (it does not keep track of the whole observed information), but we see it as an truncated version of Algorithm 30, which is itself almost admissible.

Observe that in Algorithm 30, $C_{n}(x)$ is the number of times (iterations) where the node $x$ has been crossed and $W_{n}(x)$ is the number of times where $x$ has been crossed and where this has led to a victory of $J_{1}$, all this after $n$ matches. The $(n+1)$-th match is as follows: we start from the root and make $J_{1}$ play the most promising move (for itself, i.e. the child with the highest $\phi(W_{n},C_{n})$) and $J_{0}$ play the most promising move (for itself, i.e. the child with the highest $\phi(C_{n}-W_{n},C_{n})$) until we reach an uncrossed position $z_{n}\in{\mathcal{D}}_{{{\mathbf{x}}}_{n}}$. From there, we end the match at uniform random, until we arrive at some leave $u$. We finally update the explored tree as well as its boundary and the values of the numbers of crosses and of victories of each node of the branch from $r$ to $u$.