Cooperating with Machines

TL;DR

This paper introduces a novel machine learning algorithm capable of quickly learning to cooperate with humans and other machines in various games, without complex modeling, demonstrating that flexible cooperation is computationally feasible.

Contribution

The paper presents the first general-purpose algorithm that learns to cooperate with humans in unseen environments within short timescales, without relying on complex opponent modeling.

Findings

Algorithm achieves cooperation levels rivaling humans in various games

Effective cooperation without complex opponent modeling

Fast learning in previously unanticipated game scenarios

Abstract

Since Alan Turing envisioned Artificial Intelligence (AI) [1], a major driving force behind technical progress has been competition with human cognition. Historical milestones have been frequently associated with computers matching or outperforming humans in difficult cognitive tasks (e.g. face recognition [2], personality classification [3], driving cars [4], or playing video games [5]), or defeating humans in strategic zero-sum encounters (e.g. Chess [6], Checkers [7], Jeopardy! [8], Poker [9], or Go [10]). In contrast, less attention has been given to developing autonomous machines that establish mutually cooperative relationships with people who may not share the machine's preferences. A main challenge has been that human cooperation does not require sheer computational power, but rather relies on intuition [11], cultural norms [12], emotions and signals [13, 14, 15, 16], and pre-evolved dispositions toward cooperation [17], common-sense mechanisms that are difficult to encode in machines for arbitrary contexts. Here, we combine a state-of-the-art machine-learning algorithm with novel mechanisms for generating and acting on signals to produce a new learning algorithm that cooperates with people and other machines at levels that rival human cooperation in a variety of two-player repeated stochastic games. This is the first general-purpose algorithm that is capable, given a description of a previously unseen game environment, of learning to cooperate with people within short timescales in scenarios previously unanticipated by algorithm designers. This is achieved without complex opponent modeling or higher-order theories of mind, thus showing that flexible, fast, and general human-machine cooperation is computationally achievable using a non-trivial, but ultimately simple, set of algorithmic mechanisms.

Equations

$$E(t)=\{e_j\in E:{\rho }_j(t)\ge \alpha (t)\}.$$

$$\alpha (t+m)\leftarrow {\lambda }^m\alpha (t)+(1-{\lambda }^m)R,$$

$$E(t)=\{e_j\in E_{\mathrm{cong}}(t):{\rho }_j(t)\ge \alpha (t)\}.$$

Full text

1 Introduction

The emergence of driverless cars, autonomous trading algorithms, and autonomous drone technologies highlight a larger trend in which artificial intelligence (AI) is enabling machines to autonomously carry out complex tasks on behalf of their human stakeholders. To effectively represent their stakeholders in many tasks, these autonomous machines must repeatedly interact with other people and machines that do not fully share the same goals and preferences. While the majority of AI milestones have focused on developing human-level wherewithal to compete with people [6, 7, 8, 9, 10], most scenarios in which AI must interact with people and other machines are not zero-sum interactions. As such, AI must also have the ability to cooperate, even in the midst of conflicting interests and threats of being exploited. Our goal is to understand how to build AI algorithms that cooperate with people and other machines at levels that rival human cooperation in arbitrary two-player repeated interactions.

Algorithms capable of forming cooperative relationships with people and other machines in arbitrary scenarios are not easy to come by. A successful algorithm should possess several properties. First, it must not be domain-specific – it must have superior performance in a wide variety of scenarios (generality). Second, the algorithm must learn to establish effective relationships with people and machines without prior knowledge of associates’ behaviors (flexibility). To do this, it must be able to deter potentially exploitative behavior from its partner and, when beneficial, determine how to elicit cooperation from a (potentially distrustful) partner who might be disinclined to cooperate. Third, when associating with people, the algorithm must learn effective behavior within very short timescales – i.e., within only a few rounds of interaction (learning speed). These requirements create many technical challenges (see SI.A.2), the sum of which often causes AI algorithms to fail to cooperate, even when doing so would be beneficial in the long run.

In addition to these computational challenges, human-AI cooperation is difficult due to differences in the way that humans and machines reason. While AI relies on computationally intensive search and random exploration to generate strategic behavior, human cooperation appears to rely on intuition [11], cultural norms [12], emotions and signals [13, 14], and pre-evolved dispositions toward cooperation [17]. In particular, cheap talk (i.e., costless signals) is important to human cooperation in repeated interactions [15, 16], as it helps people coordinate quickly on desirable equilibrium and create shared representations [18, 19, 20, 21]. As such, in addition to generating effective strategic behavior, a fourth important property of a successful algorithm designed for repeated interactions is the ability to communicate effectively at levels conducive to human understanding.

2 Results

The primary contribution of this work is the development and analysis of a new learning system that couples a state-of-the-art machine-learning algorithm with novel mechanisms for generating and responding to signals. Via extensive simulations and user studies, we show that this learning system learns to establish and maintain effective relationships with people and other machines in a wide-variety of repeated interactions at levels that rival human cooperation. In so doing, we also investigate the algorithmic mechanisms that are responsible for its success.

2.1 An Algorithm that Cooperates with People and Other Machines

Over the last several decades, algorithms for generating strategic behavior in repeated games have been developed in many disciplines, including economics, evolutionary biology, and the AI and machine-learning communities. To begin to evaluate the ability of these algorithms to forge successful cooperative relationships, we selected and evaluated 25 representative algorithms from these fields, including classical algorithms such as (generalized) generous tit-for-tat (i.e., Godfather) and win-stay-lose-shift (WSLS) [39], evolutionarily evolved memory-one and memory-two stochastic strategies [40], machine-learning algorithms (including reinforcement learning), belief-based algorithms [41], and expert algorithms [42, 43]. Via extensive simulations, we compared these algorithms with respect to six different performance metrics (see SI.B.2) across the periodic table of 2x2 games [37] (see Methods and SI.A.3).

The results of this evaluation, which are overviewed in Methods (see Figure 6 in particular) and are described in detailed in SI.B, demonstrate the difficulty of developing algorithms that can forge effective long-term relationships in many different scenarios. The results show that only S++ [43] was a top-performing algorithm across all metrics at all game lengths when associating with other algorithms. However, despite its fast learning speeds and its success in interacting with other machines in many different scenarios, S++ does not, in its current form, consistently forge cooperate relationships with people (SI.D and SI.E.1-2), though it does cooperate with people as frequently as people cooperate with each other in the same studies. Thus, none of these existing algorithms establishes effective long-term relationships with both people and machines.

We hypothesized that S++’s inability to consistently learn to cooperate with people appears to be tied to its inability to generate and respond to costless signals. Humans are known for their ability to effectively coordinate on cooperative equilibria using costless signals called cheap talk [15, 16]. However, while signaling comes naturally to humans, the same cannot be said of sophisticated AI algorithms, such as machine-learning algorithms. To be useful, costless signals should be connected with behavioral processes. Unfortunately, most machine-learning algorithms have low-level internal representations that are often not easily expressed in terms of high-level behavior, especially in arbitrary scenarios. As such, it is not obvious how these algorithms can be used to generate and respond to costless signals at levels that people understand.

Fortuitously, unlike typical machine-learning algorithms, the internal structure of S++ provides a clear, high-level representation of the algorithm’s dynamic strategy that can be described in terms of the dynamics of the underlying experts. Since each expert encodes a high-level philosophy, S++ could potentially be used to generate signals (i.e., cheap talk) that describe its intentionality. Speech acts from its partner can also be compared to its experts’ philosophies to improve its expert-selection mechanism. In this way, S++ can be augmented with a communication framework that gives it the ability to generate and respond to cheap talk. The resulting new algorithm, dubbed S# (pronounced ‘S sharp’), is depicted in Figure 1 (see Methods and SI.C for details about the algorithm).

We conducted a series of three user studies (see SI.D–F for details) involving 220 participants, who played in a total of 472 games, to determine the ability of S# to forge cooperative relationships with people. Representative results are found in the final (culminating) study, in which participants played three representative repeated games (drawn from distinct payoff families; see SI.A.3) via a computer interface that hid the identity of their partner. In some conditions, players could engage in cheap talk by sending messages at the beginning of each round via the computer interface. Consistent with prior work investigating cheap talk in repeated games [16], messages were limited to the predetermined speech acts available to S#.

The proportion of mutual cooperation achieved by Human-Human, Human-S#, and S#-S# pairings are shown in Figures 2a-b. When cheap talk was not permitted, Human-Human and Human-S# pairings did not frequently result in cooperative relationships. However, across all three games, the presence of cheap talk doubled the proportion of mutual cooperation experienced by these two pairings. While S#’s speech profile was distinct from that of humans (Figure 2c), subjective, post-interaction assessments indicate that S# used cheap talk to promote cooperation as effectively as people (Figure 2d). In fact, many participants were unable to distinguish S# from a human player (Figure 2e). Together, these results illustrate that, across the games studied, the combined behavioral and signaling strategies of S# were as effective as those of human players.

2.2 Distinguishing Algorithmic Mechanisms

Why is S# so successful in forging cooperative relationships with both people and other algorithms? Are its algorithmic mechanisms fundamentally different from those of other algorithms for repeated games? We have identified three algorithmic mechanisms responsible for S#’s success. Clearly, Figure 2 demonstrates that the first of these mechanisms is S#’s ability to generate and respond to relevant signals people can understand, a trait not present in previous learning algorithms designed for repeated interactions. These signaling capabilities expand S#’s flexibility in that they also allow S# to more consistently forge cooperative relationships with people. Without this capabilities, it does not consistently do so. Figure 3a demonstrates one simple reason that this mechanism is so important: cheap talk helps both S# and humans to more quickly experience mutual cooperation with their partners.

Second, our implementation of S# uses a rich set of experts that includes a variety of equilibrium strategies and even a simple learning algorithm (see SI.C.1). While none of these individual experts has an overly complex representation (e.g., no expert remembers the full history of play), these experts are more sophisticated than those traditionally considered (though not explicitly excluded) in the discussion of expert algorithms [22, 23, 24]. This more sophisticated set of experts permits S# to adapt to a variety of partners and game types, whereas algorithms that rely on a single strategy or a less sophisticated set of experts are only successful in particular kinds of games played with particular partners [25] (Figure 3c). Thus, in general, simplifying S# by removing experts from this set will tend to limit the algorithm’s flexibility and generality, though doing so will not always negatively impact its performance when paired with particular associates in particular games.

Finally, S#’s somewhat non-conventional expert-selection mechanism (see Eq. 1) is central to its success. While techniques such as $\varepsilon$-greedy exploration (e.g., EEE) and regret-matching (e,g., Exp3) have permeated algorithm development in the AI community, S# instead uses an expert-selection mechanism closely aligned with recognition-primed decision making [26]. Given the same full, rich set of experts, more traditional expert-selection mechanisms establish effective relationships in far fewer scenarios than S# (Figure 3c). Figures 3a-b provide insights into why this is so. Compared to the other expert-selection mechanisms, S# has a greater combined ability to quickly establish a cooperative relationship with its partner (Figure 3a) and then to maintain it (Figure 3b), a condition brought about by S#’s tendency to not deviate from cooperation after mutual cooperation has been established (i.e., loyalty).

The loyalty brought about by S#’s expert-selection mechanism helps explain why S#-S# pairings substantially outperformed Human-Human pairings in our study (Figure 2a-b). S#’s superior performance can be attributed to two human tendencies. First, while S# did not typically deviate from cooperation after successive rounds of mutual cooperation (Figure 3b), many human players did. Almost universally, such deviations led to reduced payoffs to the deviator. Second, a sizable portion of our participants failed to keep some of their verbal commitments. On the other hand, since S#’s verbal commitments are derived from its intended future behavior, it typically carries out the plans it proposes. Had participants followed S#’s strategy in these two regards, Human-Human pairings would have performed nearly as well, on average, as S#-S# pairings (Figure 3d – see SI.F.4 for details).

2.3 Repeated Stochastic Games

The previous results were demonstrated for normal-form games. However, S++ also learns effectively in repeated stochastic games [27], which are more complex scenarios in which a round consists of a sequence of moves by both players. In these games, S++ is distinguished, again, by its ability to adapt to many different machine associates in a variety of different scenarios [27]. As in normal-form games, S++ can be augmented with cheap talk to form S#. While S++ does not consistently forge effective relationships with people in these more complex scenarios, our results show that S# does. Representative results are shown in Figure 4, which considers a turning-taking scenario in which two players must learn how to share a set of blocks. Like people, S# uses cheap talk to substantially increase its payoffs when associating with other people in this game (Figure 4b). These results mirror those we observe in normal-form games (compare Figures 4b and 2b). See SI.E for additional details and results.

3 Discussion

Our studies of human-S# partnerships were limited to five repeated games, selected carefully to represent different classes of games from the periodic table of games (see SI.A.3). Though future work should address more scenarios, S#’s success in establishing cooperative relationships with people in these representative games, along with its consistently high performance across all classes of 2x2 games and various repeated stochastic games [27] when associating with other algorithms, gives us some confidence that these results will generalize to other scenarios.

Since Alan Turing envisioned Artificial Intelligence, major milestones have focused on defeating humans in zero-sum encounters [6, 7, 8, 9, 10]. However, in many scenarios, successful machines must cooperate with, rather than compete against, humans and other machines, even in the midst of conflicting interests and threats of being exploited. Our work demonstrates how autonomous machines can learn to establish cooperative relationships with people and other machines in repeated interactions. We showed that human-machine and machine-machine cooperation is achievable using a non-trivial, but ultimately simple, set of algorithmic mechanisms. These mechanisms include computing a variety of expert strategies optimized for various scenarios, a particular meta-strategy for a particular meta-strategy for selecting experts to follow, and the ability to generate and respond to simple signals. We hope that this first extensive demonstration of human cooperation with autonomous machines in repeated games will spur significant further research that will ensure that autonomous machines, designed to carry out human endeavors, will cooperate with humanity.

4 Methods

Detailed methods and analysis are provided in the SI. In this section, we overview three different aspects of these methods and analysis: the benchmark of games used to compare algorithms and people, results from our comparison of AI algorithms, and a description of S#.

4.1 Benchmark Games for Studying Cooperation

As with all historical grand challenges in AI, it is important to identify a class of benchmark problems to compare the performance of different algorithms. When it comes to human cooperation, a fundamental benchmark has been $2\times 2$, general-sum, repeated games [28]. This class of games has been a workhorse for decades in the fields of behavioral economics [29], mathematical biology [30], psychology [31], sociology [32], computer science [33], and political science [34]. These fields have revealed many aspects of human cooperative behavior through canonical games, such as the Prisoner’s Dilemmas, Chicken, Battle of the Sexes, and the Stag Hunt. Such games, therefore, provide a well-established, extensively studied, and widely understood benchmark for studying the capabilities of machines to develop cooperative relationships.

The periodic table of $2\times 2$ games (Figure 5; see SI.A.3; [28, 35, 36, 37, 38]) identifies and categorizes 144 unique game structures that present many unique scenarios in which machines may need to cooperate. We use this set of game structures as a benchmark against which to compare the abilities of algorithms to cooperate. Successful algorithms should be able to forge successful relationships with both people and machines across all of these repeated games. In particular, we can use these games to quantify the abilities of various state-of-the-art machine learning algorithms to satisfy the aforementioned properties: generality across games, flexibility across opponent types (including humans), and speed of learning.

Like the majority of work in repeated interactions, we focus on two-player normal-form games to more easily understand how machines can forge cooperative relationships with people. Nevertheless, we are interested in algorithms that can also be used in more complex interactions, including the more general case of repeated (two-player) stochastic games (see, for example, Figure 4). Studies evaluating the ability of S# to forge cooperative relationships with people in repeated stochastic games have yielded similar results to those we report for two-player normal-form games (e.g., Figure 4b). These studies are described in SI.E.

4.2 Interacting with Other Machines: AI Algorithms for Repeated Interactions

With the goal of identifying successful algorithmic mechanisms for playing arbitrary repeated games, we selected and evaluated 25 existing algorithms (see Figure 6a) with respect to six different performance metrics (see SI.B.2) across the periodic table of 2x2 games. These representative algorithms included classical algorithms such as (generalized) generous tit-for-tat (i.e., Godfather) and win-stay-lose-shift (WSLS) [39], evolutionarily evolved memory-one and memory-two stochastic strategies [40], machine-learning algorithms (including reinforcement learning), belief-based algorithms [41], and expert algorithms [42, 43].

Results of this evaluation are summarized in Figure 6a. Detailed analysis is provided in SI.B. We make two high-level observations. First, it is interesting to observe which algorithms were less successful in these evaluations. For instance, while generalized tit-for-tat, WSLS, and memory-one and memory-two stochastic strategies (e.g., Mem-1 and Mem-2) are successful in prisoner’s dilemmas, they are not consistently effective across the full set of 2x2 games. These algorithms are particularly ineffective in longer interactions, as they do not effectively adapt to their associate’s behavior. Additionally, algorithms that minimize regret (e.g., Exp3 [22], GIGA-WoLF [23], and WMA [24]), which is the central component of world-champion computer poker algorithms [9], also performed poorly.

Second, while many algorithms had high performance with respect to some measure, only S++ [43] was a top-performing algorithm across all metrics at all game lengths. Furthermore, it maintained this high performance in each class of game and when associating with each class of algorithm (see SI.B.5). S++ learns to cooperate with like-minded associates, exploit weaker competition, and bound its worst-case performance (Figure 6b). Perhaps most importantly, whereas many machine-learning algorithms do not learn cooperative behavior until after thousands of rounds of interaction (if at all), S++ tends to do so within relatively few rounds of interaction (Figure 6c), likely fast enough to support interactions with people.

4.3 S#: A Machine-Learning Algorithm that Talks

S# is derived from S++ [43], an expert algorithm that combines and builds on decades of research in computer science, economics, and the behavioral and social sciences. S++ uses the description of the game environment to compute a diverse set of experts, each of which uses distinct mathematics and assumptions to produce a strategy over the entire space of the game. S++ then uses a meta-level control strategy based on aspiration learning [44, 45, 46] to dynamically reduce this set of experts. Formally, let $E$ denote the set of experts computed by S++. In each epoch (beginning in round $t$), S++ computes the potential $\rho_{j}(t)$ of each expert $e_{j}\in E$, and compares this potential with its aspiration level $\alpha(t)$ to form a reduced set $E(t)$ of experts:

[TABLE]

This reduced set consists of the experts that S++ believes could potentially produce satisfactory payoffs. It then selects one expert $e_{\rm sel}(t)\in E(t)$ using a satisficing decision rule [45, 46]. Over the next $m$ rounds, it follows the strategy prescribed by $e_{\rm sel}(t)$, after which it updates its aspiration level as follows:

[TABLE]

where $\lambda\in(0,1)$ is the learning rate and $R$ is the average payoff obtained by S++ in the last $m$ rounds. It also updates each expert $e_{j}\in E$ based on its peculiar reasoning mechanism, and then begins a new epoch.

These results demonstrate the ability of S++ to effectively establish and maintain profitable long-term relationships with machines in arbitrary repeated games. Does S++ also learn to form cooperative relationships with people?

S# differs from S++ in two ways. First, the partner’s proposed plans, signaled via speech acts, are used to further reduce the set of experts that S# considers selecting (Figure 1c). Formally, let $E_{\rm cong}(t)$ denote the set of experts in round $t$ that are congruent with the last joint plan proposed by S#’s partner (see SI.C.2.2). Then, S# considers selecting experts from the following set:

[TABLE]

If this set is empty (i.e., no desirable options are congruent with the partner’s proposal), $E(t)$ is calculated as in the original S++ (Eq. 1). Second, S# also extends S++ by generating speech acts that convey the “stream of consciousness” of the algorithm (Figure 1d). Specifically, S# generates a finite-state machine with output for each expert. Given the state of the expert and the game outcomes, the state machine of the currently selected expert produces speech derived from a predetermined set of phrases. The set of speech acts, which are largely game-generic (though some adaptations must be made for multi-stage games; see SI.E.3.4) allows S# to provide feedback to its partner, make threats, provide various explanations to manage the relationship, and propose and agree to plans.

See SI.C for an in-depth description of S#.