An Empirical Evaluation of Two General Game Systems: Ludii and RBG

TL;DR

This paper empirically compares Ludii and RBG, two emerging general game systems, focusing on their simplicity and efficiency to evaluate their suitability for AI research in game-playing.

Contribution

It provides the first experimental comparison of Ludii and RBG, highlighting their relative strengths in simplicity and computational efficiency.

Findings

Ludii and RBG differ significantly in efficiency.

Ludii offers greater human-readability.

Both systems show potential for AI research.

Abstract

Although General Game Playing (GGP) systems can facilitate useful research in Artificial Intelligence (AI) for game-playing, they are often computationally inefficient and somewhat specialised to a specific class of games. However, since the start of this year, two General Game Systems have emerged that provide efficient alternatives to the academic state of the art -- the Game Description Language (GDL). In order of publication, these are the Regular Boardgames language (RBG), and the Ludii system. This paper offers an experimental evaluation of Ludii. Here, we focus mainly on a comparison between the two new systems in terms of two key properties for any GGP system: simplicity/clarity (e.g. human-readability), and efficiency.

Tables

Table 1. TABLE I: An experimental comparison between GDL, RBG, and Ludii (rate = Ludii / RBG).

								RBG	RBG		Rate	Rate
Game	GDL	RBG	Ludii	Rate			GDL	Interpreter	Compiler	Ludii	Interpreter	Compiler
	Number of tokens								Playouts per second
Amazons	1158	195	51	0.26			185	307	625	4,349	14.17	6.96
Arimaa	$\times$	735	359	0.49			$\times$	0.01	0.11	714	71,400	6,490.9
Breakthrough	670	134	65	0.49			1,123	4,962	16,694	4,741	0.96	0.28
Chess	4392	641	186	0.29			0.06	79	714	720	9.11	1.01
Chinese Checkers	$\times$	418	243	0.58			$\times$	232	1090	1,105	4.76	1.01
Connect-4	751	155	31	0.20			13,664	41,897	84,124	94,077	2.25	1.12
English Checkers	1282	263	161	0.61			872	2,963	14,286	8,135	2.75	0.57
Double Chess	$\times$	790	202	0.26			$\times$	6	50	81	13.5	1.62
Gomoku	514	324	21	0.06			927	1,330	2212	42,985	32.32	19.43
Hex	$\times$	245	81	0.33			$\times$	2741	5787	11,077	4.04	1.91
International Checkers	$\times$	498	244	0.49			$\times$	262	1,941	3,444	13.15	1.77
Reversi	894	311	103	0.33			203	1468	2,012	2,081	1.42	1.03
The Mill Game	$\times$	296	103	0.15			$\times$	1,063	7,423	72,734	68.42	9.80
Tic-Tac-Toe	381	101	25	0.25			85,319	139,312	400,000	535,294	3.84	1.34

Full text

I Introduction

Playing games has been such a long standing aspect of human culture that it can certainly be considered an integral part of our history. Strategic games can provide a complex challenge for even the most skilled human player, often requiring a precise and forethought sequence of response actions to win. As an alternative to creating specialised algorithms for specific games like Chess [1], or Go [2], developing artificial agents capable of playing a broad variety of games is the goal of General Game Playing (GGP) [3]. The modern era of GGP was launched in 2005 with the annual International General Game Playing Competition (IGGPC) announced by the Stanford’s Logic Group [4].

The General Game System GGP-BASE 111GGP-BASE https://github.com/ggp-org/ggp-base using the Game Description Language (GDL) [5] has become the standard for academic research in GGP. Games in GDL are described in terms of simple instructions based on first-order logic clauses, designed for deterministic games with perfect information. Extensions to this language also allow for imperfect information and epistemic games [6, 7]. The general intelligence required by GDL agents has led to several important research contributions [8], with one of the most popular and effective techniques being Monte Carlo tree search [9, 10], while the last winner of the IGGPC uses an approach based on constraint programming and symmetry detection [11].

However, defining games using GDL currently has several problems. Writing and debugging game descriptions requires an adept understanding of first order logic, making such tasks difficult for people who are not computer scientists or mathematicians. The equipment and rules for each game are also typically interconnected, and the structural aspects of each game, such as the board, deck, or arithmetic operators, must be explicitly defined from scratch each time. This makes game creation a lengthy and time-consuming process. Even small modifications to existing games, such as changing the size of the board, require many lines of code to be changed or added. Processing such descriptions is also computationally expensive as it requires logic resolution, making the language difficult to integrate with other external applications and limiting the potential of GDL outside of game AI.

Due to these limitations of GDL, an alternative GGP language is useful which allows games to be defined in an easier and human understandable manner, whilst also being efficient enough to facilitate future AI research in many other areas of GGP, like developing universal agents, learning, procedural content generation, or game analysis. Apart from AI research, developing such software can provide an efficient tool for work in related fields such as game design, history, or education.

II Two Alternative General Game Systems

Recently, two alternatives to GDL have been proposed: the Regular Boardgames language (RBG) [12], and Ludii [13].

II-A Regular Boardgames (RBG)

The idea behind RBG comes from an initial work proposed by [14] for using a regular language to encode the movement of pieces for a small subset of chess-like games called Simple Board Games. However, as the allowed expressions are simplistic and applied only to one piece at a time, it cannot express any non-standard behavior. RBG [12] extended and updated this idea to be able to describe the full range of deterministic board games.

The RBG system uses a low-level language given as an input for programs (agents and game manager), which is easy to process, and a high-level language, which allows for more concise and human readable descriptions. The high level version can be converted to the low level version in order to provide the two main aspects of a GGP system: human-readability and efficiency for AI programs. The technical syntax specification of RBG can be found in [15]. Thanks to this distinction between two languages, it is possible to model complex games (e.g. Amazons, Arimaa or Go), and apply the more common AI methods (minimax, Monte-Carlo search, Reinforcement Learning, etc.) to them. Indeed, in the previous languages used for GGP (including the academic state of the art GDL), it was difficult to model any complex games and impossible to play or reason on any of them in a reasonable amount of time. As an example, Figure 1 presents a complete description of Amazons for the RBG system. RBG has been demonstrated to be universal for the class of finite deterministic games with full information, and more efficient than the state of the art using GDL [12].

II-B Ludii

Within the context of the Digital Ludeme Project222Digital Ludeme Project: www.ludeme.eu [16] – which aims to model the world’s traditional strategy games in a single, playable digital database – using GDL as the game description language to model games was not feasible due to the inherent complexity of modelling games with first-order logic, and the difficulty of integrating this language with other external applications. For this reason, a new General Game system based on Browne’s thesis [17] and the notion of ludemes, called Ludii [13] was implemented.

Ludemes are the conceptual elements of a game. In Ludii, games are composed of ludemes in order to distinguish the game’s form (rules and equipment) and its function (its emergent behaviour). An important benefit of the ludemic approach is that it encapsulates key game concepts and gives them meaningful labels. Thanks to this property, it is possible to use the class grammar approach [18] to derive the game description language directly from the class hierarchy of the underlying source code library used by the system. For this reason, the game description language used by Ludii is automatically generated from the constructors in the class hierarchy of the Ludii source code. Game descriptions expressed in the grammar are automatically instantiated back into the corresponding library code for compilation, giving a guaranteed 1:1 mapping between the source code and the grammar.

This approach produces an efficient tool to model, play and analyse any strategical games as a structure of ludemes. Moreover, it is also possible to associate any kind of information like the kind of game state (e.g. stacking, boardless, stochastic, etc) to each ludeme in order to optimise the reasoning on games and to provide a playable interface for each game. Ludii makes its programming language the game description language. It can theoretically support any rule, equipment or behaviour that can be programmed, but the implementation details are hidden from the user, who only sees the simplified grammar which summarises the code to be called. A more complete description of the Ludii system is given in [13], and an example, Figure 2 presents a complete description of Amazons for Ludii.

Like RBG, Ludii has been demonstrated to be universal for the class of finite deterministic games with full information, and more efficient than GDL [13]. However, due to the novelty of the two new GGP systems, they were not yet compared to each other. We remedy this in the next section.

III Experiments

The two new GGP Systems – as most other GGP systems – use MCTS as the core method for AI move planning, which has proven to be a superior approach for general games in the absence of domain specific knowledge [10]. MCTS playouts require fast reasoning engines to achieve the desired number of simulations. Hence, we use flat Monte Carlo playouts as the metric for comparing the efficiency between Ludii and RBG.

In order to make a comparison between the human-readability of the two systems we use two main criteria: clarity and simplicity. Clarity refers to the degree to which game descriptions would be self-explanatory to non-specialist readers, and simplicity refers to the ease with which game descriptions can be created and modified, and can be estimated by the number of tokens required to define games.

III-A Setup

In the following comparison, we compare RBG and Ludii based on the numbers of tokens used to describe each game, and the number of random playouts obtained per second by each of them. All experiments were conducted on a single core of an Intel(R) Core(TM) i7-8650U CPU @ 1.90 GHz, 2112 MHz with 16GB RAM, spending 10 min per test. The RBG system is implemented in C++ and compiled with g++ 8.3 and Ludii is implemented in Java and compiled with the Java SE Development Kit 11. The RBG system provides an interpreter and a compiler to perform reasoning, consequently we compare Ludii to each of them.

III-B Results

The results of our experiments for the same set of games implemented on the two systems are shown in Table I. The left section of the table is dedicated to the number of tokens used to describe the games, and the right section to the number of playouts obtained per second (the highest number of playouts obtained for each game is coloured in blue). The rightmost columns of each section shows the rate of Ludii on RBG. Note that results are also given for GDL in order to highlight the gap between it and the new alternatives.

IV Discussion

IV-A Clarity and Simplicity

We argue that describing a game with the ludemic approach is typically simpler than using regular expressions as in RBG. Indeed, the number of tokens used by Ludii to describe a game is always much smaller than RBG; the highest rate is 0.61 for English Checkers and the lowest rate is 0.06 for Gomoku. On average Ludii needs a third of the number of tokens used by RBG to model a game.

In Ludii, the Java classes that define each ludeme use convenient definitions for the concepts involved, which provide meaningful names for each class and are aligned with human perception. For example, Container sub-classes include Board, Hand, Deck, etc., move rules include Slide, Step, etc., and game properties include stacking, hidden information, etc. These are common terms that most game players and designers would recognise and understand. In RBG, the high-level version allows predefined functions to generate regular shaped boards, and it is possible to define a game using only meaningful names with the assistance of some basic macros.

While the clarity of a game description is subjective, we believe that most – in particular non-expert users – would find Ludii game descriptions to be more clear in general than RBG descriptions. See Figures 1 and 2 for a comparison of the game descriptions for Amazons. In RBG, high-level concepts such as the movement of a queen can be summarised in a #queenShift macro, but the macro is game-specific and its low-level details are still present in the same game description. These low-level details can be difficult to understand, especially without prior knowledge of regular expressions. Similar details are completely hidden in the Ludii game description, with the move rules being encoded in ludemes with easily-understandable names such as “slide” and “shoot”. Additionally, there is a clear demarcation between starting rules, playing rules, and end conditions in Ludii.

IV-B Efficiency

Ludii outperforms RBG in terms of playouts per second in all evaluated games, except for Breakthrough (where RBG’s interpreter and compiler are both more efficient), and English Checkers (where only RBG’s compiler performs better).

For the majority of the games, the number of playouts per second computed by Ludii is significantly higher than by the RBG interpreter. If we focus on complex games, like Amazons, Chess or International Checkers, Ludii is at least $10$ times faster. Moreover for Arimaa, the RBG interpreter cannot compute a single playout per second, in contrast to $700$ playouts per second for Ludii. In the original RBG paper, a variant of Arimaa is used with a fixed position for all the pieces on the initial state in order to minimise the branching factor implied by that game, but Ludii does not need that to reason on that game.

The RBG compiler is more efficient than the RBG interpreter in terms of reasoning and the rate between it and Ludii is lower. When using the compiler version, RBG exceeds Ludii on English Checkers as well as Breakthrough. For all the other games, Ludii still provides speedups over the RBG compiler. The high performance of RBG on Breakthrough is likely due to the use of only basic onboard operations in RBG’s Breakthrough description.

V Conclusion

The ludemic General Game system Ludii and the Regular Boardgames system are two efficient alternatives which outperform the current standard for academic AI research into GGP in terms of reasoning and human-readability. In term of efficiency, Ludii outperforms the RBG interpreter in all evaluated games except one, and compared to the RBG compiler, it provides some better results in all evaluated games except two.

We argue that Ludii games are simpler to model, using fewer tokens to describe them than RBG. We also believe that Ludii game descriptions are easier to read, largely thanks to meaningful ludeme names, whereas RBG requires the use of game-specific macros to describe higher-level concepts.

Acknowledgment.

This research is part of the European Research Council-funded Digital Ludeme Project (ERC Consolidator Grant #771292) run by Cameron Browne at Maastricht University’s Department of Data Science and Knowledge Engineering.