Learning a Behavioral Repertoire from Demonstrations

TL;DR

This paper introduces Behavioral Repertoire Imitation Learning (BRIL), a method that learns a set of diverse behaviors from demonstrations, enabling precise control and adaptation of policies in complex tasks like StarCraft II.

Contribution

BRIL extends imitation learning by learning a conditioned policy that captures multiple behaviors, allowing for behavior modulation and improved performance over traditional IL.

Findings

Learned policy can be manipulated to express distinct behaviors.

Behavioral space constructed with PCA captures meaningful variations.

Adaptive behavior tuning surpasses baseline IL performance.

Abstract

Imitation Learning (IL) is a machine learning approach to learn a policy from a dataset of demonstrations. IL can be useful to kick-start learning before applying reinforcement learning (RL) but it can also be useful on its own, e.g. to learn to imitate human players in video games. However, a major limitation of current IL approaches is that they learn only a single "average" policy based on a dataset that possibly contains demonstrations of numerous different types of behaviors. In this paper, we propose a new approach called Behavioral Repertoire Imitation Learning (BRIL) that instead learns a repertoire of behaviors from a set of demonstrations by augmenting the state-action pairs with behavioral descriptions. The outcome of this approach is a single neural network policy conditioned on a behavior description that can be precisely modulated. We apply this approach to train a policy on 7,777 human replays to perform build-order planning in StarCraft II. Principal Component Analysis (PCA) is applied to construct a low-dimensional behavioral space from the high-dimensional army unit composition of each demonstration. The results demonstrate that the learned policy can be effectively manipulated to express distinct behaviors. Additionally, by applying the UCB1 algorithm, we are able to adapt the behavior of the policy - in-between games - to reach a performance beyond that of the traditional IL baseline approach.

Tables

Table 1. Table 1: Test accuracy and loss for IL, BRIL, and IL trained on clusters 10, 11, 30 and 32. Results show no significant difference between the IL and BRIL in terms of prediction accuracy. BRIL is, however, able to express multiple behaviors based on the additional input (see Table  2 ).

Method	Mean test accuracy	Mean test loss	# of replays
IL	$48.090\pm 0.080$	$1.775\pm 0.003$	7777
BRIL	48.167 $\pm$ 0.083	1.768 $\pm$ 0.003	7777
IL (C10)	$32.608\pm 0.417$	$2.096\pm 0.013$	189
IL (C11)	$71.326\pm 0.316$	$1.088\pm 0.008$	74
IL (C30)	$45.709\pm 0.499$	$1.976\pm 0.020$	149
IL (C32)	$45.855\pm 0.681$	$1.770\pm 0.006$	97

Table 2. Table 2: Results in StarCraft using Imitation Learning (IL) on the whole training set, IL on individual clusters (C10, C11, C30, and C32), Behavioral Repertoire Imitation Learning (BRIL) with fixed behavioral features corresponding to centroids in C10, C11, C30, and C32. Additionally, results are shown in which UCB1 selects between the four behavioral features in-between games. Each variant played 100 games against the easy Zerg bot. The nearest demonstration in the entire dataset was found based on the bot’s mean behavior (normalized army unit combination) and the distance to each cluster centroid are shown. The results demonstrate that by using certain behavioral features, the BRIL policy outperforms the traditional IL approach as well as IL on behavioral clusters.

		Distance to cluster centroid				Combat units produced
Method	Wins	C10	C11	C30	C32	Marines	Marauders	Hellions	S. Tanks	Reapers
IL	41/100	0.58	0.22	0.39	0.75	44.1 $\pm$ 50.5	0.7 $\pm$ 3.2	2.6 $\pm$ 7.6	1.7 $\pm$ 6.5	0.3 $\pm$ 1.1
IL (C10)	3/100	0.05	0.76	0.81	0.71	1.1 $\pm$ 2.3	0.1 $\pm$ 0.3	3.11 $\pm$ 6.1	0.1 $\pm$ 0.4	0.1 $\pm$ 0.33
IL (C11)	7/100	0.74	0.00	0.52	0.96	18.8 $\pm$ 38.4	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0
IL (C30)	18/100	0.76	0.21	0.31	0.79	43.5 $\pm$ 62.6	0.9 $\pm$ 5.4	0.2 $\pm$ 1.3	0.0 $\pm$ 0.2	0.2 $\pm$ 0.8
IL (C32)	0/100	0.71	0.94	0.57	0.04	0.1 $\pm$ 0.2	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	0.0 $\pm$ 0.0	9.9 $\pm$ 18.5
BRIL (C10)	27/100	0.21	0.85	0.81	0.60	2.4 $\pm$ 4.9	0.0 $\pm$ 0.0	14.6 $\pm$ 18.9	4.0 $\pm$ 5.2	0.2 $\pm$ 0.6
BRIL (C11)	76/100	0.70	0.05	0.53	0.95	81.4 $\pm$ 50.1	0.0 $\pm$ 0.1	0.2 $\pm$ 0.1	0.9 $\pm$ 2.4	0.3 $\pm$ 0.6
BRIL (C30)	47/100	0.60	0.31	0.29	0.65	41.6 $\pm$ 36.4	2.4 $\pm$ 6.7	0.7 $\pm$ 2.5	4.1 $\pm$ 7.8	0.5 $\pm$ 1.2
BRIL (C32)	16/100	0.42	0.72	0.53	0.36	7.1 $\pm$ 11.4	1.7 $\pm$ 6.5	3.2 $\pm$ 8.3	6.7 $\pm$ 9.7	0.8 $\pm$ 1.5
		Wins for each option				Combat units produced
Method	Win Rate	C10	C11	C30	C32	Marines	Marauders	Hellions	Siege Tanks	Reapers
BRIL (UCB1)	61/100	5/14	47/59	8/18	1/9	52.1 $\pm$ 47.7	0.5 $\pm$ 3.2	4.3 $\pm$ 12.5	2.5 $\pm$ 6.2	0.3 $\pm$ 1.3

Equations

$${\overline{X}}_j+C\sqrt{\frac{2\ln t}{n_j}},$$

Full text

Introduction

Deep Reinforcement learning has shown impressive results, especially for board games (?) and video games (?). However, reinforcement learning (RL) has critical shortcomings when reward signals are sparse or interaction with the environment is expensive. There are several attempts to mitigate these shortcomings, including curriculum learning (?; ?; ?), reward shaping (?), curiosity-driven exploration (?), diversification (?; ?), and Imitation Learning (?).

In this paper, we focus on Imitation Learning (IL), wherein the goal is to learn a policy from a dataset of demonstrations, possibly coming from a human, another artificial system, or a collection of different entities. IL can be combined with RL, either to kick-start the learning process with IL and then improving the policy further with RL (?) or by running both methods in parallel (?). Traditional IL techniques result in a single policy, which usually expresses an “averaged” behavior among all the behaviors present in the dataset. We see this as a major limitation of IL. It would be more desirable to instead learn a diverse set of policies, expressing all the different types of behaviors present in the dataset. Additionally, having a repertoire of different behaviors allows a system to adapt to changes when it is deployed.

Addressing the limitations of current IL methods, we present a new IL approach called Behavioral Repertoire Imitation Learning (BRIL), which is inspired by Quality-Diversity (QD) algorithms (?; ?) and RL methods that learn multiple different behaviors. In contrast to traditional optimization techniques, QD-algorithms attempt to find a diverse set of high-quality solutions rather than a single optimal solution. When QD-algorithms search in policy space, they typically discover hundreds or thousands of different policies controlled by different neural networks. BRIL instead learns a behavioral repertoire using a single model that can be manipulated to express multiple behaviors, similarly to RL algorithms that learn a single policy for multiple goals (?; ?). BRIL consists of a multi-step process (see Figure 1) wherein the experimenter: (1) extracts state-action pairs (similarly to many IL approaches), (2) designs a set of behavioral dimensions to form a behavioral space (similarly to many QD algorithms) and determines the behavioral description (coordinates in the space) for each demonstration, (3) merges the data to form a dataset of state-action-behavior triplets, and (4) trains a model to predict actions from state-behavior pairs through supervised learning. When deployed, the model can act as a policy and the behavior of the model can be manipulated by changing its behavioral input features.

BRIL is tested on the build-order planning problem in StarCraft, in which a high-level policy controls the build-order decisions for a bot that has otherwise scripted modules for low-level tasks (?; ?). As the results in this paper demonstrate, the behavior of the learned policy can be efficiently manipulated without further training. Additionally, the behavior of the learned policy can be optimized online with the Upper Confidence Bounds (UCB1) algorithm, outperforming a traditional IL approach when tested against one of the built-in bots. We believe this approach can be particularly useful when modeling human players in a game, where the policy should express the entire range of distinct behaviors instead of the average of all. We hypothesize that this property can allow a system to be more robust to exploitation, which is a concern for AI system in many games. Furthermore, BRIL could be useful in applications beyond games, such as adaptive and resilient robotics.

Background

Imitation Learning

While Reinforcement Learning (RL) deals with learning a mapping (a policy) between states and actions by interacting with an environment, in Imitation Learning (IL) a policy is learned from demonstrations. Methods based on IL, also known as Learning from Demonstration (LfD), have shown promise in the field of robotics (?; ?; ?; ?; ?) and games (?; ?; ?; ?; ?; ?; ?).

IL is a form of supervised learning, in which the goal is to learn a policy $\pi(s)$, mapping a state $s$ to a probability distribution over possible actions. In contrast to an RL problem, the agent cannot interact with the environment during training but is instead presented with a dataset $D$ of demonstrations. A demonstration $d_{j}\in D$ consists of $k_{j}$ sequential state-action pairs, where the action was taken in the state by some policy. While not a general requirement, in this paper a demonstration corresponds to an episode, i.e. starting from an initial state and ending in a terminal state. A policy derived from $D$ can be evaluated both on its ability to predict actions from states by cross-validation, similarly to a supervised classification task. Additionally, and perhaps more interestingly, the derived policy can be evaluated by measuring its performance in the environment.

IL spans many different methodologies and algorithms. In this paper, we focus on supervised learning methods that use deep neural networks optimized with stochastic gradient descent to approximate the state-action mapping $\pi(s,a)$ from $D$. Before reviewing approaches to learn diverse behavioral repertoires, the rest of this section briefly reviews prominent examples of IL within the domain of games.

AlphaGo, the first computer system to beat a professional human player in Go without handicap, is a prime example of IL using deep neural networks (?). AlphaGo consists of a combination of both IL, RL and forward search; IL was a key component in AlphaGo, while a newer version called AlphaZero did not rely on human demonstrations (?). AlphaStar was similarly the first computer system to beat a professional player in StarCraft, in which IL was a critical component and the starting point for further training through RL (?). Interestingly, another component in AlphaStar consists of a tournament between a population of diverse policies that are optimized through RL and a set of manually designed reward functions (?). Neural network-based IL approaches have also been applied to Quake II (?; ?), a build-order planning module as part of a scripted bot in StarCraft (?), and Candy Crush Saga (?). In ? (?), the state-action pairs were clustered based on their state representations with a self-organizing map (?) and a different policy was trained for each cluster. Multi-Action per time step Imitation Learning (MAIL) combines IL and RL, such that a policy is trained iteratively either from samples coming from demonstrations or samples from trajectories from the current policy (?).

Generative Adversarial Imitation Learning (GAIL) uses a Generative Adversarial Network (GAN) architecture wherein the generator is a policy producing trajectories (without access to rewards) and the discriminator has to distinguish between these and trajectories from a set of demonstrations (?).

Quality Diversity & Behavioral Repertoires

Traditional optimization algorithms aim at finding the optimal solution to a problem. Quality Diversity (QD) algorithms, on the other hand, attempt to find a set of high-performing solutions that each behave as differently as possible (?). QD-algorithms usually rely on evolutionary algorithms, such as Novelty Search with Local Competition (NSLC) (?) or MAP-Elites (?). NSLC is a population-based multi-objective evolutionary algorithm with a novelty objective that encourages diversity and a local competition objective that measures an individual’s ability to outperform similar individuals in the population. Individuals are added to an archive throughout the optimization process if they are significantly more novel than previously explored behaviors. MAP-Elites does not maintain a population throughout the evolutionary run, only an archive divided into cells that reflect the concept of behavioral niches in a pre-defined behavioral space. For example, in ? (?) different cells in the map correspond to different walking gaits for a hexapod robot. Both NSLC and MAP-Elites results in an archive of diverse and high-performing solutions. The pressure toward diversity in QD algorithms can help the optimization process escape local optima (?), while the diverse set of solutions also allows for online adaption by switching intelligently between these (?). We will describe variations of such an adaption procedure in the next section.

QD is related to the general idea of learning behavioral repertoires. Where QD-algorithms optimize towards a single quality objective by simultaneously searching for diversity, a behavioral repertoire can consist of solutions optimized towards different objectives as in the Transferability-based Behavioral Repertoire Evolution algorithm (TBR-Evolution) (?).

Bandit Algorithms & Bayesian Optimization

Given either a discrete set or a continuous distribution of options, we can intelligently decide which options to select in order to maximize the expected total return over a number of trials. To do this, we consider the discrete case as a $k$-armed bandit problem and the continuous case as a Bayesian optimization problem. In the continuous case, the problem can also be simplified to a $k$-armed bandit problem, simply by picking $k$ options from the continuous space of options.

The goal of a $k$-armed bandit problem is to maximize the total expected return after some number of trials by iteratively selecting one of $k$ arms/options, each representing a fixed distribution of returns (?). To solve this problem, one must balance exploitation (leveraging an option that has rendered high returns in the past) and exploration (trying options to gain a better estimation of their expected value). A bandit algorithm is a general solution to $k$-armed bandit problems. One of the most popular bandit algorithm is the Upper Confidence Bound 1 (UCB1) algorithm (?) that first tries each arm once and then always selecting the option that at each step maximizes:

[TABLE]

where $\overline{X}_{j}$ is the mean return when selecting option $j$ after $t$ steps, $n_{j}$ is the number of times option $j$ has been selected, and $C$ is a constant that determines the level of exploration.

This $k$-armed bandit approach can be considered as the discrete case of the more general approach of Bayesian optimization (BO), in which a continuous black-box objective function is optimized. BO starts with a prior distribution of the objective functions, which is updated based on queries to the black-box function using Bayes’ theorem. The Intelligent Trial and Error algorithm (IT&E) uses BO for robot adaptation to deal with changes in the environment by intelligently searching in the continuous behavioral space of policies found by MAP-Elites (?). In their approach, the fitness of all solutions in the behavioral space is used to construct a prior distribution of the fitness, which is also called a behavior-performance map. A Bayesian optimizer is then used to sample a point in the map, record the observed performance, and compute a posterior distribution of the fitness. This process is continued until a satisfying solution is found. IT&E could also be applied as an adaptation procedure to the policy found by BRIL, by creating a prior distribution based on some quality information of the demonstrations, e.g. player rating or win-rate. A Bayesian optimizer is then used to optimize the behavioral feature input. In this paper, we will instead simplify the adaption process to a discrete $k$-armed bandit problem with $k$ manually selected behavioral features and leave the use of Bayesian optimization techniques for future work.

Dimensionality Reduction

In this paper, the dimensionality of the behavioral space is reduced through a Principal Component Analysis (PCA). PCA is a popular dimensionality reduction method that finds a new set of dimensions, also called the principal components, from a set of data points. The number of dimensions can then be reduced by selecting only a subset of the principal components that explain the maximal variance of the data points. Reducing the dimensionality of a dataset from $n$ to $p$ dimensions, where $n\geq p$, using PCA consists of several fairly simple steps. First, the data is standardized, a covariance matrix is computed, and the eigenvectors and eigenvalues of the covariance matrix are found. These eigenvectors correspond to the principal components and the eigenvalues represent how well they each explain the variance in the data. The eigenvectors are thus sorted in descending order by their eigenvalues and the first $p$ eigenvectors are selected to form a new matrix called a feature vector with columns equal to the eigenvectors. The original dataset, structured as a matrix with the data points as columns, can then be transformed into the reduced space by multiplying it with the transpose of the feature matrix.

While we use PCA in this paper, there are several other dimensionality reduction techniques that could be explored. For instance, PCA cannot capture non-linear relationships in the data, in contrast to methods such as t-Distributed Stochastic Neighbor Embedding (t-SNE) (?) or UMAP (?). In the case where only the similarities (or dissimilarities) are known, rather than coordinates in the behavioral space, Multidimensional Scaling (MDS) can be a useful method (?). For example, if behaviors are described as sequences of actions, which cannot easily be transformed into coordinates, it is still possible to compute their similarities instead.

Universal Policies

In value-based RL, on typically learns a state value function $V_{\pi}(s)$ or a state-action value function $Q_{\pi}(s,a)$ for a policy $\pi$. Universal Value Function Approximators (UVFA) instead learn a joint distribution $V_{\pi}(s,g)$ or $Q_{\pi}(s,a,g)$ over all goals $G$ (?). UVFA can be learned using supervised learning from a training set of optimal values such as $V_{g}^{*}(s)$ or $Q_{g}^{*}(s,a)$, or alternatively, it can be learned through RL by switching between goals both when generating trajectories and when computing gradients. Hindsight Experience Replay is an extension to UVFAs, which performs an additional gradient update with the goal being replaced by the terminal state; this modification can give further improvements when it is infeasible to reach the goals (?). An extension to Generative Adversarial Imitation Learning (GAIL) augments each trajectory with a context (?), which specifies the agent’s sub-goals that can be modulated at test-time.

In our approach, we are not considering goals, but rather behaviors, with the aim of learning a universal policy $\pi(s,b)$ over states $s\in S$ and behaviors $b\in B$ in a particular behavioral space. We are thus combining the QD approach of designing a behavioral space with the idea of learning a universal policy to express behaviors in this space.

StarCraft

Video games are popular testbeds for the development and testing of AI algorithms (?). Real-Time Strategy (RTS) games, such as StarCraft, are among the hardest games for algorithms to learn as they contain a myriad of problems, such as dealing with imperfect information, adversarial real-time planning, sparse rewards, or huge state and action spaces (?). Several algorithms and bots have been built by the AI community (?; ?) to compete in tournaments such as the AIIDE StarCraft AI Competition111http://www.cs.mun.ca/~dchurchill/starcraftaicomp/, the CIG StarCraft RTS AI Competition222http://cilab.sejong.ac.kr/sc_competition/ and the Student StarCraft AI Competition333http://sscaitournament.com/.

The paper presented here deals with the problem of learning a policy for build order planning in StarCraft. Similarly to the work by ? (?), a neural network-based policy is trained using IL from state-action pairs, and the policy is then combined with a bot with scripted procedures for low-level tasks. However, the network in the work by ? (?) learns an “average” policy, while the approach introduced in this paper is able to learn a behavioral repertoire from demonstrations. The build order planning problem has also been approached with RL, thus optimizing a build order policy for a specific bot (?; ?).

In 2017, DeepMind and Blizzard released the StarCraft II Learning Environment (?), an API that allows custom bots and agents to interact with the binary of the game. Even though this learning environment has originally been designed for pure RL, it also allows data extraction thereby enabling IL from human demonstrations (?). DeepMind has approached the problem of creating a human-level AI for StarCraft II using a combination of IL and RL. First, a neural network was trained from supervised demonstrations. Then the model was used as a seed for a multi-agent RL scheme (called the AlphaStar League), in which the agents competed amongst themselves and learned from each other. This approach resulted in a series of competitive and diverse behaviors (?).

Behavioral Repertoire Imitation Learning (BRIL)

This section describes two approaches to learning behavioral repertoires using IL. We first describe how a behavioral space can be formed from demonstrations. Then we introduce a naive IL approach that first clusters the demonstrations based on their coordinates in the behavioral space, and then applies traditional IL on the clusters. Finally, BRIL is introduced, which learns a single policy augmented with a behavioral feature input rather than learning multiple policies for each behavioral cluster. BRIL consists of a number of steps which are depicted in Fig. 1.

Behavioral Spaces from Demonstrations

A behavioral space consists of a number of behavioral dimensions that are typically determined by the experimenter. For example, in StarCraft, behavioral dimensions can correspond to the ratio of each army unit produced throughout the game to express the strategic characteristics of the player. A behavioral space can require numerous dimensions to be able to express meaningful behavioral relationships between interesting solutions for a problem. Intuitively, if the problem is complex, more dimensions can give a finer granularity in the diversity of solutions. However, there is a trade-off between granularity and adaptation, as low-dimensional spaces are easier to search in. We thus propose the idea of first designing a high-dimensional behavioral space and then reducing the number of dimensions through dimensionality reduction techniques. In our preliminary experiments, it has shown beneficial to reduce the space to two dimensions, as it allows for easy visualization of the data distribution and it also seems to be a good trade-off between granularity and adaptation speed. In preliminary experiments with one-dimensional behavioral spaces, we noticed that nearby solutions could be wildly different.

Imitation Learning on Behavioral Clusters

A naive IL approach, that learns behavioral repertories, trains $n$ policies on $n$ behaviorally diverse subsets of the demonstrations. This idea is similar to the state-space clustering in ? (?), but here we cluster data points in a behavioral space instead. When a behavioral space is defined, each demonstration can be defined by a particular behavioral description (a coordinate in the $\mathbb{R}^{n}$ dimensional space), where afterwards a clustering algorithm can split the dataset into several subsets. Hereafter, traditional IL can be applied to each subset with the goal of learning one policy for each behavioral cluster. This approach creates a discrete set of policies similarly to current QD algorithms. However, it introduces a difficult dilemma: if the clusters are small, there is a risk of overfitting to these reduced training sets. On the other hand, if the clusters are large but few, the granularity of behaviors is lost.

Learning Behavioral Repertoires

QD algorithms typically fill an archive with diverse and high-quality solutions, sometimes resulting in thousands of policies stored in a single run, which increases the storage requirements in training as well as in deployment. To reduce the storage requirement, one can decrease the size of the archive, with the trade-off of losing granularity in the behavioral space. The main approach introduced in this paper, called Behavioral Repertoire Imitation Learning (BRIL), solves these issues and reduces overfitting by employing a universal policy instead, in which a single policy is conditioned on a behavioral description. In contrast to QD algorithms, the goal of BRIL is not to optimize neither quality nor diversity directly. Instead, BRIL attempts to imitate and express the diverse range of behaviors and the quality that exists in a given set of demonstrations. Additionally, BRIL produces a continuous space of policies which is potentially more expressive than a discrete set.

BRIL extends the traditional imitation learning setting through the following approach. First, behavioral characteristics of each demonstration are determined. If the dimensionality of these descriptions is large, it can be useful to reduce the space as described in the earlier section. A training set of state-action-behavior triplets is then constructed, such that the behavior is equal to the behavioral description of the corresponding demonstration. Then, a policy $\pi(s,b)$ is trained supervised on this dataset to map states and behaviors to actions. Using this approach, the training set is not reduced to small behavioral clusters.

When the trained policy is deployed, the behavioral feature input can be modulated with the goal of manipulating its behavior. The simplest approach is to fix the behavioral features throughout an episode, evaluate the episodic return, and then consider new behavioral features for the next episode. This approach should allow for episodic, or inter-game, adaptivity, which will be explored in our experiments. One could also manipulate the behavioral features during an episode e.g. by learning a meta-policy.

Experiments

This section presents the experimental results of applying BRIL to the game of StarCraft. Policies are trained to control the build order planning module of a relatively simple scripted StarCraft bot444https://github.com/njustesen/sc2bot that plays the Terran race. While the policy is trained off-line, our experiments attempt to optimize the playing strength of this bot online, in-between episodes/games, by manipulating its behavior.

Behavioral Feature Space

The behavioral space for a StarCraft build-order policy can be designed in many ways. Inspired by the AlphaStar League Strategy Map (?), the behavioral features are constructed from the army composition, such that the dimensions represent the ratios of each unit type. We achieve this by traversing all demonstrations in the data set, counting all the army unit creation events, and computing the relative ratios. Each demonstration thus has an $n$-dimensional behavioral feature description, where $n=15$ is the number of army unit types for Terran.

To form a 2D behavioral space, which allows for easier online search and analysis, we apply Principal Component Analysis (PCA). We have also experimented with the T-distributed Stochastic Neighbor Embedding (t-SNE), but for this dataset, the separation of points was less distinguishable. Fig. 2 visualizes the points of all the demonstrations in this 2D space, and Fig. 2a shows four plots where the points are colored to show the ratios of Marines, Marauders, Hellions, and Siege Tanks that was produced in the games.

Clustering

For the baseline approach that applies IL on behavioral clusters, we use density-based spatial clustering of applications with noise (DBSCAN) with $\epsilon=0.02$ and a minimum number of samples per cluster of 30. We performed a grid-search on these two parameters to find the most meaningful data separation; however, the clustering is not perfect due to the large number of outliers. The clusters are visualized in Fig. 2b, where outliers are colored black.

Prediction Accuracy

Data from StarCraft 2 replays are extracted with sc2reaper555https://github.com/miguelgondu/sc2reaper, a tool built using the StarCraft II Learning Environment inspired on the MSC Database (?). 7,777 replays of Terran vs. Zerg were processed, extracting state-action pairs every half a second resulting in a dataset of 1,625,671 state-action pairs.

These states contain an abstraction of the game state similar to the one found in ? (?). This abstraction includes: 1) the agent’s resources, supply, units and technologies, 2) a tally of the enemy’s units that have been observed, and 3) the agent’s units and technologies in progress, including how far they are from being completed.

Once the replays were post-processed for clustering, the dataset was split into training/test/validation following a 60% / 10% / 30% split per cluster. Three groups of neural networks were trained, all with three hidden layers and 256 hidden nodes per layer: (1) One baseline model trained on the whole dataset with no augmentation of behavioral features, (2) a BRIL model on the whole dataset with two extra input nodes for the augmented behavioral features (i.e. the coordinates in Fig. 2b), and (3) several cluster baseline models trained on demonstrations from their respective clusters without the augmented behavioral features.

These experiments were carried out ten times per model. Table 1 shows the mean test accuracy and mean test loss over these ten models for the baseline (IL) approach, the novel BRIL approach, and four different cluster baselines (Clusters 10, 11, 30 and 32), which were selected for their wildly different behaviors. The results show that augmenting by behavioral features has no significant effect on the test accuracy or loss. However, the next section shows how our new approach is able to express different behaviors with a single neural network.

Performance in StarCraft

We applied the trained policy models as build order modules in a scripted StarCraft II Terran bot called sc2bot (see above). It is important to note that this is a very simplistic bot with several flaws and limitations. Therefore the main goal in this paper is not to achieve human-level performance in StarCraft, but rather to test if BRIL allows us to do manipulate its behavior and enables online adaptation. The build order module, here controlled by one of our policies, is queried with a state description and returns a build order action, i.e. which building, research, or unit to produce next. The worker and building modules of the bot perform these actions accordingly, while assault, scout, and army modules control the army units. Importantly, policies we test act in a system that consists of both the bot, the opponent bot, and the game world. When we want to utilize our method for adaptation, we are thus not only adapting to the opponent but also the peculiarities of the bot itself.

We will first focus on the results of the traditional IL approach. Table 2 shows the number of wins in 100 games on the two-player map CatalystLE as well as the corresponding average behaviors (i.e. the army unit ratios). Our bot played as Terran against the built-in Easy Zerg bot. The traditional IL approach won 41/100 games. IL on behavioral clusters showed very poor performance with a maximum of 18/100 wins by the model trained on C30. Besides the number of wins, we compute the nearest demonstration in the entire data set from the average behavior, and use it as an estimate of the policy’s position in the 2D behavioral space. From the estimated point, we calculate the distance to each of the four cluster centroids. This analysis revealed that the policies trained on behavioral clusters express behaviors close to the clusters they were trained on (see the distances to the cluster centroids in Table 2). We hypothesize that the poor win rates of this naive approach are due to their training sets being too small such that the policies do not generalize to many of the states explored in the test environment.

Table 2 also shows the results for BRIL with the coordinates of the four cluster centroids as behavioral features. BRIL (C11) achieves a win rate of 76/100, thus outperforming traditional IL. These results demonstrate that, for some particular environment, the model can be tuned to achieve a higher performance than traditional IL. Analyzing the behavior of the bot with the behavioral features of C11 reveals that it performs an all-in Marine push, similarly to the behavior of the demonstrations in C11 (notice the position of C11 on Fig. 2.b and the illumination of Marines on Fig. 2.a). With the behavioral features of C30, the approach reached a higher win rate than traditional IL; however, this difference was not significant. We also notice that for both BRIL and IL on behavioral clusters, the average expressed behavior is closest to the cluster centroid that it was modulated to behave as, among the four clusters we selected. The results show that the behavior of the learned BRIL policy can be successfully controlled. However, the distances are on average larger than for IL on behavioral clusters. Figure 3 shows screenshots of typical army compositions produced by the BRIL policy with the four different behavioral features used.

Online Adaptation

The final test aims to verify that we can indeed use BRIL for online adaptation. We thus apply the UCB1 algorithm to select behavioral features from the discrete set of four options: {C10, C11, C30, C32} (i.e. the two-dimensional feature descriptions of these cluster centroids). This approach enables the algorithm to switch between behavioral features in-between games based on the return of the previous one, which is 1 for a win and 0 otherwise. The adaptive approach achieves 61/100 wins by identifying the behavior of C11 as the best option. Not surprisingly, the win rate is lower than when having the behavioral features of C11 fixed, but it outperforms traditional IL.

Discussion

We proposed two new IL methods in this paper, one which learns a policy that is trained on only one behavioral cluster of data points and one which learns a single modifiable policy on the whole dataset. Our results suggest that policies trained on small behavioral clusters overfit and are thus unable to generalize beyond the states available in the cluster. This drawback might be solved with fewer and larger clusters at the cost of losing granularity in the repertoire of policies. If data is abundant, this approach may also work better while we still suspect the same overfitting would occur. BRIL, on the other hand, is simple to implement and results in a continuous distribution of policies by adjusting the behavioral features. Additionally, the results suggest that BRIL generalizes better, most likely because it learns from the whole training set. However, that generality potentially comes with the cost of higher divergence between the expected behavior (corresponding to the behavioral input features) and the resulting behavior when tested. While an important concern, a divergence is somewhat expected since the test environment is very different from that of the training set (different maps and opponents).

Previous work showed how IL can kick-start learning before applying RL (?; ?). With BRIL, one can easily form a population of diverse solutions instead of just one, which may be a promising approach for domains with a plethora of strategic choices like StarCraft. Promising future work could thus combine BRIL with ideas from AlphaStar to automatically form the initial population of policies used in the AlphaStar League.

Conclusions

We introduced a new method called Behavioral Repertoire Imitation Learning (BRIL). By labeling each demonstration $d\in D$ with a behavior descriptor confined within a pre-defined behavioral space, BRIL can learn a policy $\pi(s,b)$ over states $s\in S$ and behaviors $b\in B$. In our experiments, a low-dimensional representation of the behavioral space was obtained through dimensionality reduction. The results in this paper demonstrate that BRIL can learn a policy that, when deployed, can be manipulated by conditioning it with a behavioral feature input $b$, to express a wide variety of behaviors. Additionally, the observed behavior of the policy resembles the behavior characterized by $b$. Furthermore, a BRIL trained policy can be optimized online by searching for optimal behavioral features in a given setting. In our experiments, a policy trained with BRIL was optimized online beyond the performance reached by traditional IL, using UCB1 to select among a set of discrete behavioral features.

Acknowledgments

Niels Justesen was financially supported by the Elite Research travel grant from The Danish Ministry for Higher Education and Science. Miguel González Duque received financial support from the Universidad Nacional de Colombia. Jean-Baptiste Mouret is supported by the European Research Council under Grant Agreement No. 637972 (project ResiBots). The work was supported by the Lifelong Learning Machines program from DARPA/MTO under Contract No. FA8750-18-C-0103.