BoxesZero: An Efficient and Computationally Frugal Dots-and-Boxes Agent

TL;DR

BoxesZero is a computationally efficient Dots-and-Boxes agent that outperforms existing agents while using fewer resources.

Contribution

BoxesZero introduces backward training and extended endgame theorems to improve learning efficiency in Dots-and-Boxes.

Findings

BoxesZero achieves AlphaZero-level ELO ratings in significantly less time.

BoxesZero won the 2024 Chinese Computer Game Competition in Dots-and-Boxes.

Extended endgame theorems improve Monte Carlo Tree Search efficiency.

Abstract

In recent years, deep reinforcement learning (DRL) has made significant progress in the field of games. A prime example is AlphaZero, which, despite the formidable capabilities showcased, deters many from exploring its potential because of its demands for substantial computational resources. In this paper, we introduce BoxesZero, a computationally frugal Dots-and-Boxes agent that can achieve a high level of performance using relatively fewer computational resources. BoxesZero utilizes a novel and insightful training approach called “backward training”, which starts by training from high-reward states near the end of the game and gradually trains earlier stages of the game. It also incorporates the domain knowledge of Dots-and-Boxes, such as endgame theorems, to accelerate the Monte Carlo Tree Search (MCTS) process. Furthermore, we extend the existing endgame theorems (which only include long chains) to encompass scenarios with 1-chains and 2-chains, providing corresponding proofs, which we refer to as the extended endgame theorems. This novel agent, BoxesZero, can achieve a high level of playing strength much faster than AlphaZero, substantially improving the model’s learning efficiency. With carefully tuned parameters and limited GPU resources, BoxesZero surpasses the strongest open-source Boxes agents, PRsboxes and DabbleBoxes. Experimental results demonstrate that BoxesZero achieves an ELO rating comparable to AlphaZero in significantly less time. Furthermore, BoxesZero won the championship in the Dots-and-Boxes category of the 2024 Chinese Computer Game Competition.

Figures

Full text

1. Introduction

Artificial intelligence has made significant strides in the field of board games, with numerous classical reinforcement learning (RL) algorithms, such as Q-learning [1], DQN [2], PPO [3], and SAC [4], achieving state-of-the-art results. However, when faced with complex strategic decision-making problems that feature vast state spaces and delayed rewards, these agents often underperform, still grappling with challenges like data sparsity and low utilization. It was not until the advent of AlphaGo [5] in 2016 that the immense potential of deep reinforcement learning(DRL) in board games was fully unveiled. Since then, a series of exceptional algorithms have emerged, including AlphaGo Zero [6], AlphaZero [7], KataGo [8], and MuZero [9]. These algorithms ingeniously integrate deep neural networks with Monte Carlo Tree Search (MCTS) [10,11], demonstrating superhuman performance in classic board games like Go, chess [12,13], and shogi [14]. In 2017, the Transformer [15,16] architecture showcased its remarkable potential in fields such as Natural Language Processing (NLP) and Computer Vision (CV) owing to its ability to efficiently capture long-range dependencies in sequential data. Consequently, researchers began exploring the integration of this multi-head attention mechanism into RL, where these architectures have shown great potential in enhancing game efficiency and optimizing decision-making processes [17,18]. Nevertheless, the aforementioned methods demand substantial GPU computational power, making them inaccessible to many researchers. We are committed to developing an agent that efficiently utilizes computational resources, reduces computational power requirements, and possesses high-level gameplay abilities.

Dots-and-Boxes (Boxes) [19] is a classic combinatorial game with unique rules and a complex strategy space and provides an ideal platform for validating and developing new algorithms. Its fairness ensures that the final score depends solely on the players’ strategic choices, making it a valuable experimental environment for exploring new algorithms. Although $[eqn]$ Boxes has been solved [20], the game’s complexity increases exponentially with the board size, rendering this approach impractical for larger boards. The diversity and challenging nature of strategies in the opening, middle game, and endgame phases demand substantial computational resources from traditional reinforcement learning and search methods, making them difficult to apply effectively. Currently, approaches for $[eqn]$ or larger boards are primarily dominated by traditional search methods (such as Alpha-beta [21]) combined with handcrafted evaluation functions and DRL methods, which require extensive computational resources. Furthermore, Boxes often require progressing to the very end to determine the winner, leading to a severe sparse reward [22] problem. Therefore, we chose Boxes to test our architecture because it is suitable for games with long time horizons and well-defined termination states.

To address the aforementioned issues of sparse rewards and high computational resource demands, we introduce a novel self-play training architecture called backward training. Our model begins learning from the states near the end of the game and gradually progresses as it learns the appropriate values for the states approaching termination. During this process, we leverage certain conditions to extract high-value nodes from MCTS for training to mitigate the challenges posed by data scarcity, primarily focusing on training the value network. Upon reaching the initial state, we cease extracting nodes from the tree and commence comprehensive reinforcement training for the policy network. Additionally, we introduce pruning and acceleration techniques for MCTS and extend the endgame theorem [23] to include cases with 1-chains and 2-chains, providing proofs for these extended endgame theorems.

To recap, the core innovations incorporated into BoxesZero are as follows:

• Addressing the Sparse Reward Problem: In traditional RL, agents often require multiple steps to obtain a reward, which increases the difficulty of learning. Our backward training method prioritizes learning the values of states near the end of the game and gradually progresses forward, significantly mitigating the sparse reward problem.
• Mitigating Data Scarcity: Typically, in traditional MCTS, only data from the root node are used for training, which may overlook high-quality nodes and necessitate a large number of simulations, thereby wasting computational resources. To address this, we introduce a constraint mechanism to filter out frequently visited high-quality nodes in MCTS for training. Experiments show that this method greatly improves the learning efficiency of the agent’s value network during the early stages of training. However, during the reinforcement training phase, we eliminate this data extraction method in the tree to enhance the policy network.
• Incorporating Domain Knowledge: In reinforcement learning, effectively utilizing prior knowledge without hindering exploration embodies the balance between exploration [24] and exploitation, which is a significant challenge. In our MCTS process, we introduce a bias constraint to balance exploration with the avoidance of unreasonable moves, focusing computational resources on common game states and improving efficiency. Additionally, we utilize forced endings (endgame theorems) to accelerate MCTS and enable it to obtain more accurate rewards.
• Solving Endgames with 1-Chains and 2-Chains: The existing endgame theorem for Boxes only applies to cases with long chains (length $[eqn]$ and long loops (length $[eqn]$ . However, in actual gameplay, cases with 1-chains and 2-chains are very common. Therefore, we provide endgame theorems and proofs involving cases with 1-chains and 2-chains. This extension significantly improves the hit rate of endgame situations in MCTS.

2. Related Work

Two primary categories of methods dominate the current landscape of game-playing artificial intelligence: one is represented by AlphaZero, which combines DRL with MCTS or Enhanced Monte Carlo Tree Search(EMCTS) [25]; the other is represented by traditional search methods, such as MiniMax [26], Proof Number Search (PNS) [27], Principal Variation Search (PVS) [28], Rapid Action Value Estimation (RAVE) [29], and others.

2.1. Deep Reinforcement Learning Algorithm for Boxes

AlphaZero [7] is an algorithm that has achieved remarkable success in DRL. By integrating self-play with deep neural networks and MCTS, AlphaZero has learned complex strategies without relying on human expert knowledge. Its innovative approach represents a significant evolution in the application of deep reinforcement learning, particularly in strategic games.

The combination of DRL and MCTS can be observed across various gaming environments, having a profound impact. DRL allows agents to learn optimal policies through interactions with their environment, while MCTS efficiently explores the decision space, balancing the exploration of new strategies with the exploitation of known successful ones. This combination empowers systems like AlphaZero to go beyond traditional methods, making substantial advancements in gameplay complexity and strategy development. One of AlphaZero’s most notable achievements is its ability to defeat top human players and established algorithmic programs in games such as Go, chess, and shogi. In the game of Go specifically, AlphaZero not only broke the long-standing dominance of human players but also discovered many innovative strategies through self-play, some of which had never been seen before by humans. This illustrates AlphaZero’s powerful learning capabilities and its potential to address complex decision-making problems in various domains.

AlphaZero utilizes a deep neural network parameterized by $[eqn]$ , defined as $[eqn]$ . For any given board position s, the network generates an action probability $[eqn]$ for each possible action a, along with a state value v. This combination of $[eqn]$ is used to guide the MCTS in selecting actions. The strategy $[eqn]$ returned by MCTS is then used to train the policy network, while the game’s final reward z is utilized to train the value network, i.e., $[eqn]$ . Unlike standard MCTS implementations, AlphaZero’s MCTS process specifically consists of four core steps:

1. Selection: During the selection phase, the algorithm begins at the root node and utilizes the Upper Confidence Bound for Trees (UCT) [30,31] formula to select child nodes along an already expanded path until it encounters a non-expanded node. The UCT formula balances exploration and exploitation by selecting the optimal child node. It is expressed mathematically as follows:

$$[eqn]$$

where $[eqn]$ represents the average value of the action taken at state s, $[eqn]$ denotes the prior probability of taking action a in state s (as provided by the neural network), $[eqn]$ is the visit count for state s, $[eqn]$ is the visit count for the action a in state s, and $[eqn]$ is a constant parameter that determines the exploration factor. This formula facilitates a trade-off between exploring less-visited paths and exploiting those believed to yield high rewards. 2. Expansion: Upon reaching a non-expanded node during the selection phase, the expansion phase commences. At this stage, the search tree is augmented by adding unexplored child nodes, thus increasing the search space and allowing for a more comprehensive exploration of possible actions. 3. Evaluation: At the newly added nodes, the neural network evaluates the current state and generates two outputs: the policy vector p (indicating the probabilities of each action) and the value v (representing the expected outcome from that state). This evaluation informs future decision-making processes and helps refine the strategy. 4. Backpropagation: After the evaluation step, the results are backpropagated from the current node to the root node along the path traversed during selection. The search counts and state value v for each node are updated based on the propagated results. This mechanism reflects the win rates or returns of the current strategy, effectively informing the model about the success of actions taken during the simulation.

Attempts to apply the AlphaZero framework to boxes have yielded less satisfactory results. The Dots-and-Boxes game always requires that you fill in all possible moves before it ends, which leads to the problem of sparse reward signals. In particular, in the early stages of the game, rewards are difficult to propagate back to the root node, which means that a large amount of data is required to train the neural network effectively. As a result, AlphaZero’s approach requires a long time to develop a reasonable level of play, and the computational resource demands are often prohibitive for most users, leading to less noticeable replication results.

2.2. Traditional Search Algorithm for Boxes

MiniMax [26] is one of the earliest and most widely applied algorithms in game theory, particularly suited for perfect information games like Boxes. The MiniMax algorithm traverses the game tree to minimize the potential loss in the worst-case scenario. When applied to Boxes, MiniMax is typically combined with the Alpha-Beta pruning technique to enhance computational efficiency by reducing the number of nodes that need to be evaluated. However, as the complexity of the game increases, the performance of MiniMax degrades significantly due to the exponential growth of nodes in the game tree, which is especially evident in strategy-intensive games like Boxes. Nevertheless, MiniMax remains a valuable reference tool in the research and development of more advanced game algorithms. In the study by Joseph K. Barker and Richard E. Korf, titled “Solving $[eqn]$ Dots-and-Boxes” [23], they proposed a solution based on the MiniMax algorithm. By constructing a complete game tree and combining Alpha-Beta pruning to enhance computational efficiency, they also effectively utilized board symmetry, applied optimization strategies for the midgame and endgame, and used transposition tables. The combination of these techniques allowed them to successfully solve the $[eqn]$ size Boxes game. However, as the size of the board increases, the method becomes less applicable, so the largest fully solved size of Boxes remains $[eqn]$ .

3. Game of Dots-and-Boxes

This section will introduce the fundamental rules of the Dots-and-Boxes game, along with key strategic principles and the endgame theorems, to build a comprehensive understanding of the game.

3.1. Basic Rules of Boxes

Boxes is a classic combinatorial game that enjoys widespread popularity across the world. The game is played on a rectangular grid of dots, where players take turns connecting two adjacent dots to form a line segment. Whenever a player completes the fourth side of a box, the player claims that box and gets to take another turn. The objective of the game is to complete more boxes than the opponent, and the player who secures the most boxes by the end of the game wins. Although the rules of the game are simple, the strategies involved are very complex, especially in the endgame phase. Fortunately, there are endgame theorems.

To better introduce the endgame theorems, we first present several special structures in Dots-and-Boxes: chain and opened chain, loop and opened loop, as shown in Figure 1. The terms “chain” and “loop” are named after their shapes. They share a common characteristic: If any move is made within a chain or loop, the opponent can claim all the boxes within it. For an opened chain $[eqn]$ or an opened loop, there are generally two strategic options:

1. Capture all the boxes in the chain or loop.
2. Capture part of the chain (all but two boxes) or part of the loop (all but four boxes), leaving the remaining boxes for your opponent.

Option 1 is referred to as “give up control” because occupying all the boxes may require the player to make an additional move. This move could potentially open up another chain or loop, thereby transferring the choice to the opponent and placing the player in a passive position. Option 2 is termed “keep control” because if the opponent occupies the remaining squares, they must also make another move, allowing us to maintain control over the remaining positions. Choosing between “keep control” and “give up control” for chains or loops involves combinatorial game theory. The two strategies mentioned above are exhibited in Figure 1. In the figure, the “open” operation refers to making a move in a chain or loop.

Next, we introduce three commonly used strategies in Boxes, which will lay the foundation for applying endgame theorems and domain knowledge to accelerate the tree search in Section 4.5:

(1) 2-chain strategy: When you need to open a 2-chain, you should always select the middle edge, the one between the other two edges. If you choose either of the other two edges, your opponent has two options: keep control or give up control. However, if you choose the middle edge, your opponent only has one option: give up control. Clearly, choosing the middle edge limits the choices of your opponent, placing them in a more disadvantageous position. Therefore, when playing against an intelligent opponent, selecting the middle edge is the optimal strategy, as illustrated in Figure 2.

(2) Equivalent Edges: In a chain $[eqn]$ or a loop $[eqn]$ , all edges are equivalent. This is because, regardless of which edge is played in a chain, the opponent faces two choices: keep control or give up control. In each case, given the same choice, the outcome in terms of boxes secured will ultimately be the same. For instance, in a 3-chain, if a player moves along any of the four edges, and if the opponent chooses to keep control, the first player will secure two boxes while the opponent secures one box. Conversely, if the opponent chooses to give up control, they will take all the boxes.

(3) Forced Move Strategy: For an opened 1-chain, always take the box and make an additional move. If you do not take the box and instead make a move elsewhere, your opponent can take the box and then make another move on the remaining board. Therefore, capturing this box is the optimal strategy. If the middle edge of a 2-chain is opened, you should capture the entire 2-chain (equivalent to two opened 1-chains). For a chain of $[eqn]$ , take $[eqn]$ boxes. If a loop of $[eqn]$ is opened, you should take $[eqn]$ boxes.

3.2. Endgame Theorems

This subsection offers a concise overview of the content related to the endgame theorems presented in the paper [23]. We provide a simple definition for the endgame: a position consisting solely of long chains $[eqn]$ and long loops $[eqn]$ , where each box has either 2 or 4 edges occupied. For simplicity, we will collectively refer to the long chain as a chain. Next, we will provide definitions for the terms “opener” and “controller” in the context of the endgame.

Opener: In your turn during the endgame, you need to choose a loop or a chain to open. The player making this choice is called the opener.

Controller: Once the opener selects a component C (loop or chain), the controller, who is the other player, then has two options: keep control or give up control.

Before delving into the endgame theorems, we define a board position as G. For instance, $[eqn]$ indicates that G consists of a 4-loop, a 6-loop, a 3-chain, two 2-chains, a 1-chain, and a 6-chain, as well as possibly some already-claimed boxes, as shown in Figure 3.The value $[eqn]$ of any endgame G refers to the margin by which the controller will defeat the opener, assuming both players enter the endgame with a tied score and play optimally. Table 1 presents the definitions of the symbols and functions, all of which are defined in relation to G. The controlled value $[eqn]$ is defined to compute $[eqn]$ , with its explicit formula given below:

$$[eqn]$$

where # represents the number of the component, and $[eqn]$ , which denotes the terminal bonus of G, is defined as follows:

$$[eqn]$$

Following that, we will present three theorems that can be directly applied to the previously defined endgame scenarios. These theorems offer insight into the number of boxes that can be won or lost in the endgame and the optimal next moves.

Theorem 1 provides the opener strategy for endgames, Theorem 2 offers the controller strategy, and Theorem 3 explains how to calculate the value of an endgame, denoted as v. By applying these three theorems, once the game reaches an endgame position, all subsequent moves can be determined entirely by the theorems. Additionally, the number of boxes won or lost can be precisely calculated, allowing for an immediate assessment of the game’s outcome upon entering the endgame phase. The three theorems are as follows:

Theorem 1. (Opener Strategy) Suppose G is a nonempty Dots-and-Boxes position that consists of loops and long chains. In each of the following cases, opening the shortest loop is optimal:

• 1. 

• $[eqn]$ 2 and $[eqn]$ 3+ (one or more loops);*

• 2. 

• $[eqn]$ 0, ±1} and $[eqn]$ 4_l_ + (anything except 3 + 3 + 3);*

• 3. 

• $[eqn]$ 2 and $[eqn]$ 4_l_ + 3 $[eqn]$ , where 4 $[eqn]$ and H has no 3-chains.*

In all other cases, the standard move is optimal.

Theorem 2. (Controller Strategy) If the opener has just opened a component C of G, the following gives an optimal move to the controller. Keep control if C is a loop and $[eqn]$ 4, or if C is a chain and $[eqn]$ 2; otherwise, give up control.

Theorem 3. (Values Procedural) Suppose G is a nonempty Dots-and-Boxes position consisting of loops and long chains. Then

• 1.  if $[eqn]$ 2, then $[eqn]$ ;
• 2.  if $[eqn]$ 0, G has a 4-loop, and $[eqn]$ 4_l_ + 3 + 3, then $[eqn]$ 0;
• 3.  if $[eqn]$ 0, or if $[eqn]$ 1 and $[eqn]$ 3 $[eqn]$ 4, then

$$[eqn]$$

In all other cases, $[eqn]$ 1 or 2 according to the parity of G.

$[eqn]$ is a subset component of G. It includes the following components of G: its loops consist of all loops of $[eqn]$ , and its chains are either all chains of $[eqn]$ (if any exist) or the 3-chain (if G contains one and no other chains). Furthermore, the symbol $[eqn]$ denotes the number of 3-chains in $[eqn]$ . The functions $[eqn]$ , $[eqn]$ , and $[eqn]$ are defined as follows: $[eqn]$ , $[eqn]$ , and $[eqn]$ . In the presented formula, $[eqn]$ indicates that the $[eqn]$ operation is applied iteratively $[eqn]$ times, where $[eqn]$ corresponds to the number of 4-loops within G. Likewise, $[eqn]$ and $[eqn]$ denote the iterative application of the $[eqn]$ and $[eqn]$ operations $[eqn]$ and $[eqn]$ times, respectively.

4. Methodology

In this section, we will detail the key techniques employed in BoxesZero, including the backward training architecture, the extension of the endgame theorem, data augmentation, and MCTS pruning.

4.1. Reinforcement Learning in BoxesZero

In BoxesZero, we also use a deep neural network $[eqn]$ with parameters $[eqn]$ , which takes the multi-channel representation of the board state s as input and outputs an action probability p and a state value estimate v, i.e., $[eqn]$ . In the MCTS search, each node s stores a visit count $[eqn]$ and a state value $[eqn]$ , while each edge $[eqn]$ maintains a visit count $[eqn]$ and an action value $[eqn]$ . Here, the notation $[eqn]$ indicates that after taking move a from position s, the simulation eventually reaches state $[eqn]$ . After that, we can calculate the state value of node $[eqn]$ using the following formula:

$$[eqn]$$

The Q referred to in the following discussion denotes the $[eqn]$ output by MCTS, while v is the value output by the network.

In training the policy network, we use the action probabilities $[eqn]$ provided by MCTS, which is consistent with the approach used by AlphaZero. At the same time, we employ a weighted combination of the Q value from the root node of the tree and the final reward z (which belongs to {1, −1}) as the reward for training the value network. This differs from AlphaZero’s method, which solely uses the final reward z. We do this primarily for the following reasons:

Firstly, assigning the same reward z to all states can introduce significant bias. While z accurately reflects the value of the last few moves, it may overlook earlier steps due to the inherent randomness in the final action selection by MCTS. If a particular choice leads to the loss of the entire game because of random factors, it can negate the value of all previous steps, which is clearly inefficient. This limitation requires the model to consume more computational resources during training.

Secondly, while z provides a binary classification of outcomes (win or loss), it lacks the detailed representation of state values necessary in complex game scenarios. Different states may correspond to varying probabilities of winning, but z fails to capture the subtle differences in value between these states. The diversity and complexity of intermediate positions highlight that relying solely on the final win-loss outcome does not fully reflect the true potential and value of these states.

On the other hand, the value Q from MCTS offers a nuanced estimation of the expected value based on the current state and moves taken, allowing for better decision-making throughout the game. However, it may be influenced by noise and estimation biases inherent in the MCTS process. Therefore, combining the final reward z with the Q values provides a more robust approach. This combination allows the model to leverage the clear win-loss signal from z to learn long-term outcomes, while also incorporating the detailed state evaluations provided by Q. By doing so, the value network can better understand the overall strategy and make informed decisions, ultimately improving performance and reducing uncertainty in the learning process.

Data collected through a new reinforcement learning algorithm, backward self-play (detailed in Section 4.2.1), is used to train and update the neural network’s parameters $[eqn]$ .

4.2. Overview of Backward Training

We introduce a novel training methodology called backward training. We divide the backward training process into two key stages: the value network reinforcement stage and the policy network reinforcement stage. The value network reinforcement stage sets a step limit and utilizes the data generated by the model after exceeding this limit for training. Once the model satisfies a predefined advancement condition, we propel the model forward. The policy network reinforcement stage commences after the model advances to step 0. At this point, we utilize data from all steps for training, with the primary objective of achieving convergence for the policy network. We will now elaborate on the backward self-play method and elucidate the model advancement conditions.

4.2.1. Backward Self-Play

During the initial training phase, AlphaZero typically uses extensive self-play to collect data for training. However, due to the model’s weak playing strength in the early stages, only the data from the last few steps are relatively accurate, while the data from the initial steps is mostly randomly generated. This leads to inconsistent training data quality, affecting the network’s convergence speed and hindering the improvement of the model’s capabilities.

To mitigate this issue, we propose an improvement to the traditional self-play paradigm, aiming to fully utilize computational resources and enable the model to achieve a high level of proficiency within a shorter timeframe. Specifically, during the initial stages of model training, we selectively learn from the data, focusing only on the last few steps that are closer to the final reward and ignoring the lower-quality data from the earlier steps. As training progresses, the model’s predictive ability gradually strengthens, and the accuracy of the data from the later steps improves accordingly. At this point, we gradually incorporate data from earlier steps into the learning process, eventually encompassing all steps.

We define a threshold, “start searching” (abbreviated as $[eqn]$ ), to determine when to begin using MCTS. The entire self-play process can be divided into two stages. The first stage involves collecting data from the end to the beginning, utilizing the Q values returned by MCTS to train the value network. During this phase, the value network converges relatively quickly. As the number of training iterations and time increase, the model’s predictive capability is enhanced, which causes the $[eqn]$ value to progress forward until it reaches step 0. At this point, we enter Stage 2, where we start collecting data anew for self-play training and use the outcome z to assist in refining the value network. The introduction of the z at this juncture facilitates the convergence of the global strategy. The reward training value network is derived by combining the z and Q with weights of 0.75 and 0.25, respectively. This particular 0.75/0.25 weighting has been empirically established as effective and is widely adopted in machine game-play scenarios.

For the move selection before the $[eqn]$ step, we employ a greedy handcrafted policy in the game of Boxes. The core idea of this policy is to minimize the opportunities for the opponent to score. The policy prioritizes filling the fourth edge of a box, followed by the first/second edges and, lastly, the third edge. In practice, most Boxes models follow this greedy strategy. However, some advanced models might intentionally leave some boxes open in the midgame to create a more favorable endgame for themselves. Since we do not learn the knowledge before $[eqn]$ , this greedy strategy does not negatively impact the model’s improvement. Moreover, the game states generated by this greedy strategy contain numerous chains and loops, which are similar to our definition of the endgame. This allows the model to encounter more endgame scenarios during MCTS, thereby enhancing its learning efficiency in the endgame phase. For other board games, if there are no applicable hand-made rules or strategies in the early stages, we can directly use the smoothed policy network values p to select moves. Furthermore, the game states generated using the above strategy are not searched; therefore, data sampling is not performed.

After reaching $[eqn]$ , each subsequent move is determined using MCTS until the game ends. The backward self-play process is illustrated in Figure 4.

4.2.2. Model Advancement Conditions

In board games, the complexity of the state of the game is typically lower in the early and late stages, whereas the midgame is the most complex. As a result, more iterations may be required during the midgame, while fewer iterations are needed in the late game. Therefore, determining whether the current step is sufficiently trained and whether it is time to move forward is critical.

We evaluate the training sufficiency of a step by statistically analyzing Q-value variations observed across multiple self-play game states within a single iteration. Notably, we observed that as training progresses, $[eqn]$ output by the MCTS gradually increases from 0 to 1. This is because, when the policy network is weak, the quality of nodes explored by MCTS is poor, making it difficult for Q to accurately reflect the expected outcome. In such cases, Q in the search tree tend to cancel each other out, as some nodes predict win and others predict lose, resulting in an average Q close to 0. As training continues and the policy network improves, MCTS is able to identify favorable nodes, and $[eqn]$ approaches 1, reflecting clearer expectations of the results. Therefore, the proximity of Q of the root to 1 can serve as an indicator of training progress.

In each iteration, we assess the neural network’s performance by calculating two proportions: $[eqn]$ and $[eqn]$ . $[eqn]$ represents the fraction of states that satisfy both $[eqn]$ and $[eqn]$ . $[eqn]$ is the proportion of positions where $[eqn]$ . (Parameters $[eqn]$ , $[eqn]$ , $[eqn]$ , and $[eqn]$ are described in Table 2.) The condition $[eqn]$ , where $[eqn]$ is the neural network’s prediction and $[eqn]$ is a more precise value, measures the network’s ability to predict the correct sign. The additional condition $[eqn]$ further refines this measurement by filtering out predictions with low confidence. Finally, the relationship between $[eqn]$ and 1 is used as an indicator of the neural network’s overall capability in outputting realistic values. If both $[eqn]$ and $[eqn]$ are larger, it indicates that our model performs better at distinguishing winning and losing positions.

Ideally, $[eqn]$ for each step would reach 1. However, in practical observations, without sufficient computational resources, it is difficult to achieve such values for the first 20 moves. If training continues for an extended period without meeting strict conditions and remains stuck in one step, it can lead to overfitting of the network. Since later steps will continue to be trained after moving forward, we need to set a maximum number of iterations that satisfy the progression conditions to prevent being stuck at a particular step.

Based on the analysis above, we present the following advancement conditions.

Advancement Conditions: The conditions for progressing consist of two criteria: a constraint on Q and v and a constraint on the number of iterations. If either of the two conditions is met, the model can advance. The specific conditions are as follows: (1) $[eqn]$ . (2) $[eqn]$ .

The “current iter” refers to the current iteration number of the model, while “last iter” indicates the iteration number where the model was during its last advancement (where one iteration refers to generating data through multiple self-play games for training). “mit” denotes the maximum iteration threshold for model advancement. The specific meanings of the variables “mqr”, “mqrth”, “nvr”, “nvrth”, and “mit” are provided in Table 2. In condition (2), all the variables are used to calculate the ratio of self-play game states that satisfy Q and v conditions within one iteration, which determines whether the model can advance.

At the end of each iteration, the model checks if the advancement conditions are met. If they are, advance the model by $[eqn]$ steps.

4.3. Data Augmentation

The utilization of a backward training strategy facilitates accelerated convergence to accurate reward estimations, leading to higher training data efficacy. This approach results in a significantly increased prevalence of high-value nodes within the search tree. Consequently, we implement conditional selection mechanisms within Monte Carlo Tree Search to strategically select such nodes for further training, thereby addressing the inherent data scarcity challenges in reinforcement learning.

As states approach the end of the game, they are closer to receiving rewards, causing the value of $[eqn]$ returned by MCTS to approach 1. Conversely, for states that are further from the end, reward propagation becomes more difficult, resulting in $[eqn]$ approaching 0. Therefore, we can assume that nodes with larger $[eqn]$ have higher quality, indicating that the network can clearly distinguish the win or loss of the state. We use this characteristic as one of the criteria for evaluating high-quality nodes. Relying solely on Q constraints is insufficient to determine the quality of a node. Therefore, we also incorporate a weak constraint based on the visit count N of the tree node.

We impose higher $[eqn]$ constraints for states closer to the end and lower constraints for states further to the end. Due to prior training, states closer to the game’s end exhibit higher quality. To maintain a balanced data distribution across different steps, these nodes require stricter selection criteria. The data augmentation method is illustrated in Figure 5, and the node selection criteria are as follows: $[eqn]$ and $[eqn]$ . The specific settings for $[eqn]$ and $[eqn]$ are provided in Table A2. After filtering the data, we applied rotational symmetry augmentation to further expand the dataset before storing it in the experience replay buffer.

During the value network reinforcement stage, we emphasize the use of this data augmentation method, as it can accelerate the convergence of the value network. However, we do not employ data augmentation during the policy network reinforcement stage. This is because the augmented data have smaller N, which can interfere with the convergence of the policy network.

4.4. Improved Endgame Theorems

The previous endgame theorem restricts its application to positions involving long chains, making it inapplicable to positions that include 1-chain and 2-chain. However, in high-level gameplay, 1-chain and 2-chain are very common. To address this, we extended the endgame theorem to apply to positions that include 1-chain and 2-chain. We proposed an open strategy and provided a formula for calculating the value. After extensive validation on positions involving 1-chains and 2-chains, we confirmed the correctness of the extended theorem.

Theorem 4. (Extended Opener Strategy) Suppose G consists of chains and loops and that there exists a 1-chain or a 2-chain. If there is at least one 1-chain, opening a 1-chain is optimal. If there is no 1-chain but a 2-chain exists, opening the 2-chain and following the 2-chain policy is optimal.

Theorem 5. (Extended Values) Suppose G consists of chains and loops, containing k 1-chains and n 2-chains. Let C represent the remaining portion after removing all the 1-chains and 2-chains. The value $[eqn]$ can be determined using Theorem 3. Then, the value $[eqn]$ is given by the following:

• 1.  If k is odd and n is odd: $[eqn]$ 1;
• 2.  If k is odd and n is even: $[eqn]$ 1 $[eqn]$ ;
• 3.  If k is even and n is odd: $[eqn]$ 2 $[eqn]$ ;
• 4.  If k is even and n is even: $[eqn]$ .

Since 1-chain and 2-chain (because of the 2-chain strategy) do not involve control strategies, there is no need to consider the control policy. The proof of the theorem is provided in Appendix C.

Figure 3 illustrates an example of the application of the theorem: $[eqn]$ , we want to calculate $[eqn]$ . According to Theorem 5 and with $[eqn]$ and $[eqn]$ , we have $[eqn]$ . By applying Theorem 4, which requires occupying all 1-chains and 2-chains, we obtain $[eqn]$ . We then calculate $[eqn]$ using Theorem 3, as $[eqn]$ and $[eqn]$ , so we choose the third case of the theorem. First, based on the definition of $[eqn]$ , we find $[eqn]$ . According to Formula (2), we have $[eqn]$ . Given $[eqn]$ , $[eqn]$ , and $[eqn]$ , we calculate $[eqn]$ as follows: $[eqn]$ . Therefore, $[eqn]$ .

4.5. Pruning in MCTS

Based on the four core steps of MCTS outlined in Section 2.1, we prune the search tree that incorporates the endgame theorems and the three strategies described in Section 3.1. This approach reduces ineffective searches in MCTS by introducing endgame theorems and chain-and-loop strategies during the expansion phase, allowing for early win/loss predictions. This enables rewards to be quickly propagated from child nodes to the root, thereby accelerating the convergence of both the policy and value networks. The specific details are as follows:

1. Endgame Theorem Pruning: During the search process, if a node’s position s meets the definition of an endgame, Theorems 3 and 5 can be directly applied to calculate the value $[eqn]$ . Taking into account the current territory situation, we can calculate the score of state s as follows: $[eqn]$ . The positive and negative values of $[eqn]$ correspond to $[eqn]$ in the MCTS nodes, thus achieving the purpose of pruning. This allows the search tree to encounter actual rewards earlier. It has been verified that for a $[eqn]$ Dots-and-Boxes game, which consists of 60 moves in total, MCTS can start detecting endgames around the 30th move, enabling effective reward propagation.
2. Chain-Loop Pruning: In the tree, if the current position contains an opened chain or opened loop, the forced-move strategy mentioned in Section 3.1 can be applied to execute multiple consecutive moves, thereby reducing the search depth of MCTS. This leads to faster encounters with the endgame, thus mitigating the issue of sparse rewards and significantly improving search efficiency and decision quality.

5. Implementation Details

5.1. System Overview

We primarily adopt a distributed system architecture, which is divided into three main components: the Computing Center, the Controlling Center, and the Training Center. The Computing Center consists of ten computers, each equipped with an Intel(R) Core(TM) i5-8500 CPU. The Controlling Center and Training Center each consist of one machine equipped with an NVIDIA GeForce RTX 3060 GPU. The Computing Center generates self-play data, the Training Center learns from a shared experience replay buffer, and the Controlling Center manages data distribution and network parameter updates. This design enables concurrent data collection and network training, significantly improving time efficiency.

The system operates as follows: The Controlling Center sends data collection tasks and neural network parameters to the ten computing nodes. All computing nodes initiate self-play simultaneously and gather data on MCTS during the process. The data collected are then sent from the computing nodes to the Controlling Center. The Controlling Center aggregates the data and sends it, along with the network parameters, to the Training Center. The Training Center updates the network parameters and sends the new neural network parameters back to the Controlling Center. The Controlling Center begins to assess whether to advance the model based on the advancement conditions (see Section 4.2.2). This process continues cyclically. The architecture of the system is illustrated in Figure 6.

5.2. Neural Network Training Process

During each training cycle, the model samples a batch of data from the replay buffer for training. The replay buffer stores the $[eqn]$ information generated by the model during different iterations, which is subsequently utilized in training. In each batch, the model performs forward propagation through the neural network on the input data to compute $[eqn]$ and finally computes the loss.

The loss function used in the training process consists of three components: policy loss, value loss, and regularization loss. The policy loss measures the difference between the model’s predicted policy and the target policy (i.e., the $[eqn]$ provided by MCTS). The value loss measures the difference between the model’s predicted value and the target value (i.e., the Q provided by MCTS). The regularization loss mitigates overfitting by penalizing large weights of the neural network parameters. To ensure that recent data are prioritized during training, a decay factor is introduced. This factor adjusts the contribution of each sample to the loss based on its iteration count: samples closer to the current iteration contribute more significantly to the loss, while older samples have diminished impact. The loss function is defined as follows:

$$[eqn]$$

In this equation, D denotes the number of recent iterations stored in the replay buffer. The terms $[eqn]$ and Q represent the true policy and value labels derived from MCTS, while $[eqn]$ and $[eqn]$ denote the policy and value predicted by the network. The decay factor for the data is represented by $[eqn]$ , $[eqn]$ signifies the neural network parameters, and $[eqn]$ is the coefficient for the regularization term $[eqn]$ . The specific values of these parameters are provided in Table A1.

5.3. Neural Network Architecture

The neural network plays a crucial role in generating the estimations for the policy distribution $[eqn]$ and the value $[eqn]$ , effectively replacing the simulation phase of the MCTS. This network is organized into three main components: the input layer, the feature extraction layer, and the output layer, as illustrated in Figure 7.

The first part, the input layer, is responsible for the initial decomposition of the board. The network assumes that the current player is on the red side. If the player is on the blue side, the board should be flipped to switch the active player to the red side before making predictions. Therefore, there is no need to create a separate channel to represent the current player in the board state; simply perform a player conversion when inputting to the network. The original board is an $[eqn]$ matrix, which we divide into three channels:

• The first channel represents the edges, where positions with edges are marked as 1, and those without edges are marked as 0.
• The second channel represents the boxes, where the positions with boxes are marked as 1, and those without boxes are marked as 0.
• The third channel represents the relative scores, calculated as follows:

$$[eqn]$$

The second part, the feature map layer, is primarily composed of parallel blocks, which further extract features from the board using Kaiming He [32] initialization to optimize the training process. Additionally, a cosine learning rate schedule, specifically through Stochastic Gradient Descent with Warm Restarts (SGDR) [33], is employed to dynamically adjust the learning rate throughout training, enhancing convergence and improving overall model performance.

The third part includes the policy head and the value head, which aggregate the features extracted by the second part and reshape them to the appropriate output.

6. Experiments

This section will introduce the experimental baseline, evaluation metrics, and environmental setup. We will validate the performance of the proposed pruning method, data augmentation, and backward training while also conducting an in-depth analysis of the characteristics of our model.

6.1. Experiment Setups

6.1.1. Baselines

We evaluate our algorithm using the following well-known algorithms with established results: (1) MiniMax [26]: We reproduced several acceleration strategies from the paper, applying pruning and hashing techniques to the $[eqn]$ Dots-and-Boxes. During the initial stages, when a complete search is not feasible, we utilized manual strategies for game-play. Our implementation of the alpha-beta search can achieve results in approximately 22 steps and within about 300 s. (2) AlphaZero [7]: We implemented the AlphaZero framework for Dots-and-Boxes, with training parameter settings as detailed in Table A1. (3) Dabble: (Dabble https://www.mathstat.dal.ca/~jpg/dabble/, accessed on 2 January 2025) is a computer program that uses alpha-beta search to play Dots-and-Boxes. Its understanding of the game is minimal; it knows about chains and double-crosses but nothing else. Despite this, Dabble is able to play at a reasonable skill level; higher-level strategies emerge from brute-force search. (4) PRBoxes: (PRBoxes https://www.dianneandpaul.net/PRsBoxes/, accessed on 2 January 2025) Uses full nimstring analysis etc.

6.1.2. Metrics

We primarily use relative ELO ratings [34] and the accuracy at specific game steps to quantify algorithm performance. To compare the capabilities of different algorithms, we select models of each algorithm at various time points and conduct several matches (with alternating starting players). This allows us to compute relative ELO scores for both A and B at each time point and subsequently plot the ELO curves.

As the midgame is crucial in Dots-and-Boxes, we constructed a dataset of various game steps using the method described in Appendix E.3. This dataset allows us to test the prediction accuracy of different models at various stages of the game, thereby reflecting their respective capabilities.

6.1.3. Experimental Environment

Our experimental environment is based on a distributed system that consists of three components: the computing center, the controlling center, and the training center. The computing center comprises 10 computers equipped with Intel(R) Core(TM) i5-8500 CPUs and no GPUs. In contrast, the control center and training center are represented by a single machine that features an NVIDIA GeForce RTX 3060 GPU.

6.2. Comparative Evaluation of Backward Training

We validate the effectiveness of the backward training method by comparing it with the traditional forward training method. We ensured that the search counts, training iterations, and learning rates were identical for both the forward model and the backward model. And neither model uses endgame theorems or acceleration methods. The key difference is that our model employs the end-to-start training method.

We conducted 1000 matches between models at different time points, and the resulting ELO scores over time are depicted in Figure 8. As shown in the figure, the ELO scores of the backward training show an upward trend over time, and in the early stages of training, its playing strength is significantly higher than that of the forward training. In later stages, the curve stabilizes but remains above the forward training. This indicates that backward training can achieve higher playing strength in a shorter period and demonstrates a certain level of stability. Therefore, we conclude that the end-to-start method achieves higher playing strength and enables the model to reach the same strength level in less time.

6.3. Analysis of Loss Function Curve

We analyzed the loss curves during the training process. Based on the loss function defined in Equation (6), we plotted the cross-entropy loss for the policy network and the mean squared error loss for the value network of AlphaZero and BoxesZero, as shown in Figure 9. From the figures, it is evident that BoxesZero exhibits a characteristic where the loss is very low in the early stages of training. This is because, at the beginning, the model encounters relatively simple positions, allowing the network to fit these scenarios quickly. However, as the model advances and encounters new positions, the network must readjust the distribution of its parameters, causing the loss to increase temporarily before declining again after some time. This cycle continues until reaching step one, at which point the loss stabilizes. In contrast, AlphaZero maintains a more stable downward trend in its loss. The policy loss curve for AlphaZero is relatively flat, while the value loss curve fluctuates significantly. Conversely, BoxesZero experiences greater fluctuations in the policy loss curve but achieves a smoother and faster convergence in the value loss curve. This indicates that BoxesZero places more emphasis on training the value function, whereas AlphaZero focuses more on the convergence of the policy function. The overall downward trend of the curves demonstrates that our model is effective. Moreover, the behavior of the curves explains why our model can achieve strong playing strength more rapidly.

6.4. Ablation Studies

To validate the effectiveness of each acceleration strategy, we conducted an ablation study by decomposing our three core methods: the endgame theorem, chain-loop pruning in MCTS, and data augmentation. We assessed their individual impact on overall model performance by comparing the relative ELO ratings of four groups trained under identical parameters (30 h total training time, with models saved at 30 time steps). Baseline: Utilizes the endgame theorem, chain-loop pruning and data augmentation simultaneously. No Endgame: Removes the endgame theorem from the baseline. No Chain Loop: Removes chain-loop pruning from the baseline. No Data Augmentation: Removes data augmentation from the baseline.

We played 1000 games between every pair of models and presented the resulting relative ELO curves in Figure 10. To further analyze performance, we used the models saved at 10, 20, and 30 h to predict outcomes on the datasets described in Section 6.1.2, with the accuracies presented in Table 3. This table highlights the predictive capabilities of the different models at various training durations. The results demonstrate the effectiveness of both our pruning methods and data augmentation.

As shown in Figure 10, the “No Chain Loop” group exhibits the lowest relative ELO score (around 1100 points), indicating that chain-loop pruning provides the most significant performance improvement. Data augmentation follows, with the endgame theorem having the least impact.

Furthermore, the figure reveals synergistic effects between our methods. The combination of the endgame theorem and chain-loop pruning yields notably better results than either method alone. This is likely because chain-loop pruning leads to more reasonable game positions, increasing the likelihood of encountering endgame scenarios and, thereby, amplifying the impact of the endgame theorem. While the endgame theorem’s effect may be relatively minor on a 6 × 6 board, its significance will likely increase on larger boards due to the formation of longer chains and loops.

6.5. Evaluation of Model’s Predictive Capabilities at Different Steps

We evaluate the algorithms of BoxesZero and AlphaZero at multiple time points using datasets described in Section 6.1.2. We also include additional random moves for testing. The corresponding prediction accuracies are provided, as shown in Table 4. The orange-marked data in the table indicate the value of parameter $[eqn]$ for BoxesZero at the respective current time. For instance, when training reaches 10 h, the parameter $[eqn]$ , which indicates that training has progressed to step 26.

Through this table, we can clearly see that BoxesZero continues to improve while not forgetting previous knowledge. This is because the old knowledge is reinforced during the advancement, leading to a continuous increase in prediction accuracy.

Specifically, BoxesZero’s prediction accuracy is much higher than that of AlphaZero, especially when training reaches 60 h and 90 h, where the accuracy reached $[eqn]$ . This reflects the advantages of backward training, as it can fully utilize the real returns to achieve high-level performance with low computational resources.

6.6. Analysis of BoxesZero and AlphaZero Data Distributions

We analyze the contribution of Data Augmentation to the data volume and the characteristics of data generated by backward self-play. We present heatmaps of the data volume of BoxesZero and AlphaZero under different steps and iterations, as shown in Figure 11, and 3D surface plots, as shown in Figure 12. This allows us to visually observe the contribution of our data augmentation method to the training data volume, as well as the data distribution characteristics of backward self-play and forward self-play.

Specifically, from the data heatmap of BoxesZero in Figure 11, we can see the obvious data distribution characteristics of BoxesZero. At the beginning, there is only data after $[eqn]$ steps. As the iteration increases, the data distribution gradually moves forward. The data heatmap of AlphaZero is almost the same before 40 steps, which is its characteristic: it uses the same amount of data for each step. After 40 steps, the data distribution is less, possibly because one side has already reached 13 points (there are 25 boxes in the $[eqn]$ Boxes game) and won, and no more data are generated. From the color depth of the two heatmaps, we can see that the data volume of BoxesZero is larger than that of AlphaZero, which largely alleviates the problem of data scarcity in reinforcement learning. Moreover, the data volume of Boxes in the earlier steps is larger than that in the later steps, showing a decreasing trend. This reflects that backward training focuses on learning newly encountered situations in each step, and fewer situations are used for review in later steps, which can improve the learning efficiency of the model. Figure 12 analyzes the data distribution from a three-dimensional perspective, and the data advantage of BoxesZero can be clearly seen from the z-axis.

Overall, our method alleviates the problem of data scarcity in reinforcement learning to a certain extent.

6.7. Final Performance

To evaluate the final playing strength of BoxesZero, we set the training parameters as illustrated in Table A1. The advance parameters are presented in Table 2. Ultimately, our model utilized a distributed training framework that ran for 10 days, conducted 60,000 self-play games, and incorporated rotation symmetry and data augmentation methods. The feature extraction layer of the neural network consists of 6 parallel blocks with approximately 3 million parameters. In the MCTS, we integrated endgame theorems and employed chain-loop pruning techniques.

We evaluated our trained model using the baseline model (see Section 6.1.1). We conducted matches for each pair of models and calculated the corresponding relative ELO ratings. The results are presented in Figure 13. Observing the results, we note that our model was able to defeat the strongest open-source agent Dabble’s algorithm at 130 h of training $[eqn]$ . At 110 h of training $[eqn]$ , it surpassed the PRBboxes algorithm. Additionally, during the 40 to 50-hour training period, it outperformed both the MiniMax algorithm and the AlphaZero agent trained for 240 h. This indicates that BoxesZero’s playing strength improves faster. Moreover, it is crucial to emphasize that under the aforementioned configuration and after 240 h of training, AlphaZero’s Elo score is indeed lower than that of traditional search algorithms such as Dabble. However, with increased training duration and computational resources, AlphaZero’s performance ceiling will surpass that of traditional search algorithms. Regarding the performance ceiling of BoxesZero and AlphaZero, we believe there is no significant difference between them. This is because when BoxesZero’s backward self-play enters stage 2, its only distinction from the AlphaZero framework lies in the different labels of the value network, which are $[eqn]$ and z, respectively.

Finally, upon reaching step 0, we reinforced the policy training. After 15 days of training, we developed our strongest version. Our program ultimately won the championship in the Dots-and-Boxes category of the 2024 Chinese Computer Game Competition.

7. Conclusions

We present BoxesZero, an efficient and computationally frugal Dots-and-Boxes agent. We employ a novel backward training approach and extract high-quality nodes within the MCTS process, effectively utilizing rewards and computational resources while mitigating the issue of sparse rewards. Furthermore, we leverage the domain knowledge of Dots-and-Boxes to prune the MCTS, accelerating the convergence of the neural network. Concurrently, we provide and prove endgame theorems encompassing 1-chain and 2-chain scenarios, which were previously unproven. By combining this innovative training method with a domain knowledge-pruned MCTS, BoxesZero achieves strong performance with low computational requirements.

Experimental results demonstrate that BoxesZero requires less time and computational resources than AlphaZero to achieve the same level of performance, and it outperformed the strongest open-source programs available online, including Dabble and PRsboxes. Ultimately, we integrated BoxesZero with the endgame theorems to achieve a championship in the Dots-and-Boxes category of the 2024 Chinese Computer Game Competition.

In summary, we showcase the advantages of backward training, namely its full utilization of rewards and reduction of unnecessary computational resource waste. This architecture can also be transferred to other games, enabling anyone to train a powerful agent using their own computer. Moreover, we contribute to the game of Dots-and-Boxes by completing its endgame theorems, filling a previous gap in the field. Furthermore, we seamlessly integrate domain knowledge with MCTS, enhancing the balance between exploration and exploitation.

However, our model also has limitations. For instance, the control conditions for model advancement are manually set based on repeated observations of the training process, which presents challenges for parameter tuning. Finally, we hope that our model will inspire other areas of artificial intelligence and anticipate further validation of its performance in other reinforcement learning environments.