Near-Optimal Online Egalitarian learning in General Sum Repeated Matrix Games

TL;DR

This paper introduces an algorithm for two-player repeated games with unknown rewards that guarantees near-optimal regret bounds, leading to improved bargaining solutions over traditional maximin strategies.

Contribution

The paper presents a novel algorithm achieving near-optimal regret bounds for egalitarian bargaining in general sum repeated games with unknown rewards.

Findings

Achieves high-probability regret bound of order O(ln T^{1/3} T^{2/3})

Proves a lower bound of Ω(T^{2/3}) showing near-optimality

Demonstrates improved bargaining outcomes over maximin strategies

Abstract

We study two-player general sum repeated finite games where the rewards of each player are generated from an unknown distribution. Our aim is to find the egalitarian bargaining solution (EBS) for the repeated game, which can lead to much higher rewards than the maximin value of both players. Our most important contribution is the derivation of an algorithm that achieves simultaneously, for both players, a high-probability regret bound of order $\mathcal{O}(\sqrt[3]{\ln T}\cdot T^{2/3})$ after any $T$ rounds of play. We demonstrate that our upper bound is nearly optimal by proving a lower bound of $\Omega(T^{2/3})$ for any algorithm.

Tables

Table 1. (a) Game

	C	D
C	$\frac{4}{5},\frac{4}{5}$	$\frac{1}{10},\frac{9}{5}$
D	$\frac{9}{5},0$	$\frac{3}{10},\frac{3}{10}$

Table 2. (a) Game

	C	D
C	$\frac{4}{5},\frac{4}{5}$	$\frac{1}{10},\frac{9}{5}$
D	$\frac{9}{5},0$	$\frac{3}{10},\frac{3}{10}$

Table 3. (b) Comparison of solutions

Maximin	Egalitarian	Nash	Sum	Correlated	Pareto
$\frac{3}{10},\frac{3}{10}$	$\frac{23}{25},\frac{23}{25}$	$\frac{3}{10},\frac{3}{10}$	$\frac{1}{10},\frac{9}{5}$	$\frac{3}{10},\frac{3}{10}$	$\frac{9}{5},0$

Table 4. Table 2 : Lower bounds example. The rewards are generated from a Bernoulli distribution whose parameter is specified in the table. The first value in parentheses is the one for the first player while the other is for the second player. Here, ϵ italic-ϵ \epsilon is a small constant defined in the proof.

	$a_1^2$	$a_2^2$	$\cdots$	$a_{\left|{𝒜}^2\right|}^2$
$a_1^1$	$(0.5,1)$	$(0.5,0.5)$	$\cdots$	$(0.5,0.5)$
$a_2^1$	$(0.5\left[+ϵ\right],0.5\left[+ϵ\right])$	$(0.5,0.5)$	$\cdots$	$(0.5,0.5)$
$⋮$	$⋮$	$⋮$	$\cdots$	$(0.5,0.5)$
$a_{\left|{𝒜}^1\right|}^1$	$(0.5,0.5)$	$(0.5,0.5)$	$\cdots$	$(0.5,0.5)$

Equations

$$V_M^i(\pi )$$

$${\pi }_{\text{S}V}^i$$

$$x{\ge }_{\mathrm{\ell }}y⟺\left(L^1(x)>L^1(y)\right)\vee \left(L^1(x)=L^1(y)\wedge L^2(x)\ge L^2(y)\right)\text{for any}x\in {\mathcal{R}}^2,y\in {\mathcal{R}}^2$$

$${\text{Regret}}_T(\Lambda ,{\pi }^{-i})=\sum _{t=1}^T\text{S}{V}^i-r_t^i$$

$${\text{Regret}}_T(\Lambda ,{\Lambda }^{\prime })$$

$${\mathcal{M}}_k=\{r:|E{r}^i(a)-{\bar{r}}_k^i(a)|\le C_k(a)\forall i,a\},C_k(a)=\sqrt{\frac{2\ln 1/{\delta }_k}{N_{t_k}(a)}},$$

$${\hat{r}}_k^i(a)={\bar{r}}_k^i(a)+C_k(a),{\check{r}}_k^i(a)={\bar{r}}_k^i(a)-C_k(a).$$

$$V_{{\tilde{M}}_k}({\tilde{\pi }}_k){\ge }_{\mathrm{\ell }}{V}_{M^{\prime }}({\pi }^{\prime })\forall {\pi }^{\prime },M^{\prime }\in {\mathcal{M}}_k\mid \mathrm{Pr}\left\{V_{M^{\prime }}({\pi }^{\prime })\ge V_M({\pi }_{\text{Eg}})-({ϵ}_k,{ϵ}_k)\right\}=1$$

$$\text{S}{V}^i-{ϵ}_k\le \text{S}{\check{V}}^i\le \text{S}{V}^i+{ϵ}_k\forall i$$

$$w(a,a^{\prime })=\{\begin{array}{ll}0,& \text{if }{\hat{r}}_{+}^i(a)\le {\hat{r}}_{+}^{-i}(a)\text{ and }{\hat{r}}_{+}^i(a^{\prime })\le {\hat{r}}_{+}^{-i}(a^{\prime })\\ 1,& \text{if }{\hat{r}}_{+}^i(a)\ge {\hat{r}}_{+}^{-i}(a)\text{ and }{\hat{r}}_{+}^i(a^{\prime })\ge {\hat{r}}_{+}^{-i}(a^{\prime })\\ \frac{{\hat{r}}_{+}^{-i}(a^{\prime })-{\hat{r}}_{+}^i(a^{\prime })}{\left({\hat{r}}_{+}^i(a)-{\hat{r}}_{+}^i(a^{\prime })\right)+\left({\hat{r}}_{+}^{-i}(a^{\prime })-{\hat{r}}_{+}^{-i}(a)\right)},& \text{otherwise }\end{array}$$

$${\tilde{\pi }}_{k,\text{Eg}}(a_{\text{Eg}})=w(a_{\text{Eg}},a_{\text{Eg}}^{\prime });{\tilde{\pi }}_{k,\text{Eg}}(a_{\text{Eg}}^{\prime })=1-w(a_{\text{Eg}},a_{\text{Eg}}^{\prime });a_{\text{Eg}},a_{\text{Eg}}^{\prime }=\underset{a\in \mathcal{A},a^{\prime }\in \mathcal{A}}{\mathrm{argmax}}\text{score}(a,a^{\prime })$$

$${\tilde{\mathcal{A}}}_i$$

$${\hat{a}}_i$$

$$\tilde{\mathcal{P}}=\{i\in \{1,2\}\mid {\hat{r}}_{+}^i({\hat{a}}^i)>{\hat{V}}_{+}^i({\hat{\pi }}_{k,\text{Eg}})\}$$

$${\text{Regret}}_T$$

$$+2\sqrt{\sqrt{T|\mathcal{A}|}}+6+\sqrt{\frac{T}{2}\cdot \ln \left(\frac{T}{{\delta }^4}\cdot \frac{45}{99.375-{\pi }^4}\right)}\le O\left(12\sqrt[3]{|\mathcal{A}|\ln T}\cdot T^{2/3}\right)$$

$${\text{Regret}}_T$$

$$+2\sqrt{\sqrt{T|\mathcal{A}|}}+6+\sqrt{\frac{T}{2}\cdot \ln \left(\frac{T}{{\delta }^4}\cdot \frac{45}{99.375-{\pi }^4}\right)}\le O\left(12\sqrt[3]{|\mathcal{A}|\ln T}\cdot T^{2/3}\right)$$

$$E$$

$$E_1$$

$$E_2$$

$$E_3$$

$$E_4$$

$${\text{Regret}}_T^i$$

$$=\sum _{t=1}^T{I}_{E=1}\left(V_{\text{Eg}}^i-r_t^i\right)+\sum _{t=1}^T{I}_{E=0}\left(V_{\text{Eg}}^i-r_t^i\right)$$

$$\le \sum _{t=1}^T{I}_{E=1}+\sum _{t=1}^T{I}_{E=0}\left(V_{\text{Eg}}^i-r_t^i\right)$$

$${\text{Regret}}_T^i$$

$${\Delta }_k=\sum _{a\in \mathcal{A}}{N}_k(a)\left(V_{\text{Eg}}^i-E{r}^i(a)\right)$$

$${\Delta }_{k,\neg E}$$

$$=\sum _{a\in \mathcal{A}}{N}_k(a)\left({\hat{V}}_k^i-{\hat{r}}^i(a)\right)+\sum _{a\in \mathcal{A}}{N}_k(a)\left({\hat{r}}^i(a)-E{r}^i(a)\right)+4\sum _{a\in \mathcal{A}}{N}_k(a){ϵ}_{t_k}$$

$$=\sum _{a\in \mathcal{A}}{N}_k(a)\left({\hat{V}}_k^i-{\hat{r}}^i(a)\right)+4\sum _{a\in \mathcal{A}}{N}_k(a){ϵ}_{t_k}+\sum _{a\in \mathcal{A}}{N}_k(a)\left({\bar{r}}^i(a)-E{r}^i(a)+\frac{C_r(t_k)}{\sqrt{N_{t_k}(a)}}\right)$$

$$\le \sum _{a\in \mathcal{A}}{N}_k(a)\left({\hat{V}}_k^i-{\hat{r}}^i(a)\right)+4\sum _{a\in \mathcal{A}}{N}_k(a){ϵ}_{t_k}+\sum _{a\in \mathcal{A}}\frac{2C_r(t_k)N_k(a)}{\sqrt{N_{t_k}(a)}}$$

$$\le |\mathcal{A}|+4\sum _{a\in \mathcal{A}}{N}_k(a){ϵ}_{t_k}+\sum _{a\in \mathcal{A}}\frac{2C_r(t_k)N_k(a)}{\sqrt{N_{t_k}(a)}}$$

$$\sum _{k=1}^m{\Delta }_k$$

$$=m|\mathcal{A}|+4\sum _{k=1}^m{ϵ}_{t_k}{N}_k+\sum _{k=1}^m\sum _{a\in \mathcal{A}}\frac{2C_r(t_k)N_k(a)}{\sqrt{N_{t_k}(a)}}$$

$$\sum _{k=1}^m{ϵ}_{t_k}{N}_k$$

Full text

1 Introduction

Multi-agent systems are ubiquitous in many real life applications such as self-driving cars, games, computer networks, etc. Agents acting in such systems are usually self-interested and aim to maximize their own individual utility. To achieve their best utility, agents face three key fundamental questions: exploit, cooperate, or insure safety? Learning how agents should behave when faced with these significant challenges is the subject of this paper. We focus on two players (one is called agent, the other opponent) repeated games, a setting which captures the key challenges faced when interacting in a multi-agent system where at each round, the players simultaneously select an action and observe an individual numerical value called reward. The goal of each player in this game, is to maximize the sum of accumulated rewards over many rounds. One of the key dilemmas for learning in repeated games is the lack of a single optimal behavior that is satisfactory against all opponents, since the best strategy necessarily depends on the opponent.

Powers et al., [24] tackle this dilemma and propose a rigorous criterion called guarded optimality which for two players simplifies to three criteria: (1) Targeted Optimality: when the opponent is a member of the target set, the average reward is close to the best response against that opponent; (2) Safety: against any opponent, the average reward is close to a safety value; (3) Individual Rationality: in self-play, the average reward is Pareto efficient111i.e., it is impossible for one agent to change to a better policy without making the other agent worse off. and individually not below the safety value.

In this paper, we adopt those criteria and focus on the self-play settings. We pick the safety value to be the largest value one can guarantee against any opponent (also called maximin value, see Definition 5). For the individual rationality criterion, we depart from previous works by considering the so called egalitarian bargaining solution (EBS) [17] in which both players bargain to get an equal amount above their maximin value. This EBS is a Nash equilibrium (NE) for the repeated game, a direct consequence of the folk theorems [22] and in many games (see Example 1) has a value typically no worse for both players than values achievable by single-stage (i.e. non-repeated) NE usually considered in the literature. We pick the EBS since it satisfies even more desirable properties [17] on top of the individual rationality criterion such as: independence of irrelevant alternatives (i.e. eliminating choices that were irrelevant does not change the choices of the agents), individual monotonicity (a player with better options should get a weakly-better value) and (importantly) uniqueness. It is also connected to fairness and Rawls, [26] theory of justice for human society [17].

Related work

Our work is related to Munoz de Cote and Littman, [19] where an algorithm to find the same egalitarian solution for general-sum repeated stochastic games is provided. When applied to general-sum repeated games, their algorithm implies finding an approximate solution using a (binary) search through the space of policy. Instead, our result will find the exact egalitarian solution with a more direct and simple formula. Also Munoz de Cote and Littman, [19] and many other works such as [18, 24, 7] assume deterministic rewards known to both players. In this work, we consider the case of stochastic rewards generated from a fixed distribution unknown by both players.

Another difference with many previous works is the type of solution considered in self-play. Indeed, we consider a NE for the repeated game whereas works such as [7, 8, 3, 23] consider the single-stage NE. The single-stage NE is typically undesirable in self-play since equilibrium with much higher values can be achieved as illustrated by Example 1 in this paper. Other works such as [24] consider optimizing for the sum of rewards in self-play. However, as illustrated by Example 1 in this paper, this sum of rewards does not always guarantee individual rationality since some player could get lower than their maximin.

Crandall and Goodrich, [9], Stimpson and Goodrich, [27] proposes algorithms with the goal of converging to a NE of the repeated games. However, [9] only show an asymptotic convergence empirically in a few games while [27] only show that some parameters of their algorithms are more likely to converge asymptotically. Instead, we provide finite-time theoretical guarantees for our algorithm. Although in their settings players only observe their own rewards and not the other player, they assume deterministic rewards.

Wei et al., [28], Brafman and Tennenholtz, [5] tackles online learning for a generalization of repeated games called stochastic games. However, they consider zero-sum games where the sum of the rewards of both players for any joint-action is always 0. In our case, we look at the general sum case where no such restrictions are placed on the rewards. In the learning settings there are other single-stage equilibrium considered such as correlated-equilibrium [13].

Our work is also related to multi-objective multi-armed bandit [11] by considering the joint-actions as arms controlled by a single-player. Typical work consider on multi-objective multi-armed bandit tries to find any solution that minimizes the distance between the Pareto frontier. However, not all Pareto efficient solutions are acceptable as illustrated by Example 1 in this paper. Instead, our work show that a specific Pareto efficient (the egalitarian) is more desirable.

Paper organization

The paper is organized as follows: Section 2 presents formally our setting, assumptions, as well key definitions needed to understand the remainder of the paper. Section 3 shows a description of our algorithm while section 4 contains its analysis as well as the lower bound. We conclude in section 5 with indication about future works. Detailed proof of our theorems is available in the appendix.

2 Background and Problem Statement

We focus on two-player general sum repeated games. At round $t$, both players select and play a joint action $a_{t}=(a^{i}_{t},a^{-i}_{t})$ from a finite set $\mathcal{A}=\mathcal{A}^{i}\times\mathcal{A}^{-i}$. Then, they receive rewards $(r_{t}^{i},r_{t}^{-i})\in[0,1]^{2}$ generated from a fixed but unknown bounded distribution depending on their joint action. The actions and rewards are then revealed to both players. We assume the first agent to be under our control and the second agent to be the opponent. We would like to design algorithms such that our agent’s cumulative rewards are as high as possible. The opponent can have one of two types known to our agent: (1) self-player (another independently run version of our algorithm) or (2) arbitrary (i.e any possible opponents with no access to the agent’s internal randomness).

To measure performance, we compare our agent to an oracle that has full knowledge of the distribution of rewards for all joint-actions. The oracle then plays like this: (1) in self-play, they both compute before the game start the egalitarian equilibrium and play it; (2) against any other arbitrary opponent, the oracle plays the policy ensuring the maximin value.

Our goal is to design algorithms that have low expected regret against this oracle after any number of $T$ rounds, where regret is the difference between the value that the oracle would have obtained and the value that our algorithm actually obtained. Next, we formally define the terms that describe our problem setting.

Definition 1** (Policy).**

A policy $\pi^{i}$ in a repeated game for player $i$ is a mapping from each possible history to a distribution over its actions. That is: $\forall t\geq 0,\pi^{i}:\mathcal{H}_{t}\to\Delta\mathcal{A}^{i}$ where $t$ is the current round and $\mathcal{H}_{t}$ is the set of all possible history of joint-actions up to round $t$.

A policy is called stationary if it plays the same distribution at each round. It is called deterministic stationary if it plays the same action at each round.

Definition 2** (Joint-Policy).**

A joint policy $(\pi^{i},\pi^{-i})$ is a pair of policies, one for each player $i,-i$ in the game. In particular, this means that the probability distributions over actions of both players are independent. When each component policy is stationary, we call the resulting policy stationary and similarly for deterministic stationary.

Definition 3** (Correlated-Policy).**

*Any joint-policy where player actions are not independent is correlated222For example through a public signal.. A correlated policy $\pi$ specifies a probability distribution over joint-actions known by both players: $\forall t\geq 0,\pi:\mathcal{H}_{t}\to\Delta\mathcal{A}$. *

In this paper, when we refer to a policy $\pi$ without any qualifier, we will mean a correlated-policy, which is required for the egalitarian solution. When we refer to $\pi^{i}$ and $(\pi^{i},\pi^{-i})$ we will mean the components of a non-correlated joint-policy.

2.1 Solution concepts

In this section, we explain the two solution concepts we aim to address: safety–selected as the maximin value and individual rationality selected as achieving the value of the EBS. We start from the definition of a value for a policy.

Definition 4** (Value of a policy).**

The value $V^{i}(\pi)$ of a policy $\pi$ for player $i$ in a repeated game $M$ is defined as the infinite horizon undiscounted expected average reward given by:

[TABLE]

We use $V_{M}=(V_{M}^{i},V_{M}^{-i})$ to denote values for both players and drop $M$ when clear from the context.

Definition 5** (Maximin value).**

The maximin policy $\pi_{\scriptscriptstyle\text{S}\!V}^{i}$ for player $i$ and its value $\text{S}\!V^{i}$ are such that:

[TABLE]

where $V^{i}(\pi^{i},\pi^{-i})$ is the value for player $i$ playing policy $\pi^{i}$ while all other players play $\pi^{-i}$.

Definition 6** (Advantage game and Advantage value).**

Consider a repeated game between two players $i$ and $-i$ defined by the joint-actions $\mathcal{A}=\mathcal{A}^{i}\times\mathcal{A}^{-i}$ and the random rewards $r$ drawn from a distribution $R:\mathcal{A}\to\Delta\mathbb{R}^{2}$. Let $\text{S}\!V=(\text{S}\!V^{i},\text{S}\!V^{-i})$ be the maximin value of the two players. The advantage game is the game with (random) rewards $r_{+}$ obtained by subtracting the maximin value of the players from $r$. More precisely, the advantage game is defined by: $r_{+}(a)=r(a)-\text{S}\!V\;\forall a\in\mathcal{A}$. The value of any policy in this advantage game is called advantage value.

Definition 7** (EBS in repeated games).**

Consider a repeated game between two players $i$ and $-i$ with maximin value $\text{S}\!V=(\text{S}\!V^{i},\text{S}\!V^{-i})$. A policy $\pi_{\scriptscriptstyle\text{Eg}}$ is an EBS if it satisfies the following two conditions: (1) it belongs to the set $\Pi_{\text{Eg}}$ of policies maximizing the minimum of the advantage value for both players. (2) it maximizes the value of the player with highest advantage value.

More formally, for any vector $x=(x^{1},x^{2})\in\mathbb{R}^{2}$, let $L:\mathbb{R}^{2}\to\mathbb{R}^{2}$ be a permutation of $x$ such that $L^{1}(x)\leq L^{2}(x)$. Let’s define a lexicographic maximin ordering $\operatorname*{\geq_{\ell}}$ on $\mathbb{R}^{2}$ as:

[TABLE]

A policy $\pi_{\text{Eg}}$ is an EBS 333Also corresponds to the leximin solution to the Bargaining problem [4]. if: ${V(\pi_{\text{Eg}})-\text{S}\!V}\operatorname*{\geq_{\ell}}{V(\pi)-\text{S}\!V}\;\forall\pi$

We call EBS value the value $V_{\scriptscriptstyle\text{Eg}}=V(\pi_{\text{Eg}})$ and $V_{+}(\pi_{\text{Eg}})=V(\pi_{\text{Eg}})-\text{S}\!V$ will be used to designate the egalitarian advantage.

2.2 Performance criteria

We can now define precisely the two criteria we aim to optimize.

Definition 8** (Safety Regret).**

The safety regret for an algorithm $\Lambda$ playing for $T$ rounds as agent $i$ against an arbitrary opponent $\pi^{-i}$ with no knowledge of the internal randomness of $\Lambda$ is defined by:

[TABLE]

Definition 9** (Individual Rational Regret).**

The individual rational regret for an algorithm $\Lambda$ playing for $T$ rounds as agent $i$ against its self $\Lambda^{\prime}$ identified as $-i$ is defined by:

[TABLE]

Example 1** (Comparison of the EBS value to other concepts).**

In Table 1, we present a game and give the values achieved by the single-stage NE, and Correlated Equilibrium [13] (Correlated); maximizing the sum of rewards (Sum), and a Pareto-efficient solution (Pareto). In this game, the maximin value is $(\frac{3}{10},\frac{3}{10})$. Sum leads to $\frac{1}{10}$ for the first player, much lower than its maximin. Pareto is also similarly problematic. Consequently, it is not enough to converge to any Pareto solution since that does not necessarily guarantee rationality for both players. Both NE and Correlated fail to give the players a value higher than their maximin while the EBS shows that a high value $(\frac{23}{25},\frac{23}{25})$ is achievable. A conclusion similar to this example can also be made for all non trivial zero-sum games.

3 Methods Description

Generic structure

Before we detail the safe and individual rational algorithms, we will describe their general structure. The key challenge is how to deal with uncertainty, the fact that we do not know the rewards. To deal with this uncertainty, we use the standard principle of optimism in the face of uncertainty [16]. It works by a) constructing a set of statistically plausible games containing the true game with high probability through a confidence region around estimated mean rewards, a step detailed in section 3.1; b) finding within that set of plausible games the one whose EBS policy (called optimistic) has the highest value, a step detailed in section 3.2; c) playing this optimistic policy until the start of an artificial epoch where a new epoch starts when the number of times any joint-action has been played is doubled (also known as the doubling trick), a step described in Jaksch et al., [16] and summarized by Algorithm 3 in Appendix G.

3.1 Construction of the plausible set

At epoch $k$, our construction is based on creating a set $\mathcal{M}_{k}$ containing all possible games with expected rewards $\operatorname{\mathbb{E}}r$ such that,

[TABLE]

where $N_{t_{k}}(a)$ is the number of times action $a$ has been played up to round $t_{k}$, $\bar{r}_{k}(a)$ is the empirical mean reward observed up to round $t_{k}$ and $\delta_{k}$ is an adjustable probability. The plausible set can be used to define the following upper and lower bounds on the rewards of the game:

[TABLE]

We denote $\hat{M}$ the game with rewards $\hat{r}$ and $\check{M}$ the game with $\check{r}$. Values in those two games are resp. denoted $\hat{V}$, $\check{V}$. We used $C_{k}(\pi)$, $C_{k}(\pi^{i},\pi^{-i})$ to refer to the bounds obtained by a weighted (using $\pi$) average of the bounds for individual action. When clear from context, the subscript $k$ is dropped.

3.2 Optimistic EBS policy

§1 Problem formulation

Our goal is to find a game $\tilde{M}_{k}$ and a policy $\tilde{\pi}_{k}$ whose EBS value is near-optimal simultaneously for both players. In particular, if we refer to the true but unknown game by $M$ and assume that $M\in\mathcal{M}_{k}$ we want to find $\tilde{M}_{k}$ and $\tilde{\pi}_{k}$ such that:

[TABLE]

where $\operatorname*{\geq_{\ell}}$ is defined in Definition 7 and $\epsilon_{k}$ a small configurable error.

Note that the condition in (2) is required (contrarily to single-agent games [16]) since in general, there might not exist a game in $\mathcal{M}_{k}$ that achieves the highest EBS value simultaneously for both players. For example, one can construct a case where the plausible set contains two games with EBS value (resp) $(\frac{1}{2}+\epsilon,\frac{1}{2}+\epsilon)$ and $(\frac{1}{2},1)$ for any $0<\epsilon<1$ (See Table 2 in Appendix E). This makes the optimization problem (2) significantly more challenging than for single-agent games since a small $\epsilon$ error in the rewards can lead to a large (linear) regret for one of the player. This is also the root cause for why the best possible regret becomes $\Omega(T^{2/3})$ rather than $\Omega(\sqrt{T})$ typical for single agent games. We refer the this challenge as the small $\epsilon$-error large regret issue.

§2 Solution

To solve (2), a) we set the optimistic game $\tilde{M}_{k}$ as the game $\hat{M}$ in $\mathcal{M}_{k}$ with the highest rewards $\hat{r}$ for both players. Indeed, for any policy $\pi^{\prime}$ and game $M^{\prime}\in\mathcal{M}_{k}$, one can always get a better value for both players by using $\hat{M}$; b) we compute an advantage game corresponding to $\tilde{M}_{k}$ by estimating an optimistic maximin value for both players, a step detailed in paragraph §3; c) we compute in paragraph §4 an EBS policy $\tilde{\pi}_{k,\text{Eg}}$ using the advantage game; d) we set the policy $\tilde{\pi}_{k}$ to be $\tilde{\pi}_{k,\text{Eg}}$ unless one of three conditions explained in paragraph §5 happens. Algorithm 2 details the steps to compute $\tilde{\pi}_{k}$ and to correlate the policy, players play the joint-action minimizing their observed frequency of played actions compared to $\tilde{\pi}_{k}$ (See function $\textsc{Play}($) of Algorithm 3 in Appendix G).

§3 Optimistic Maximin Computation

Satisfying (2) implies we need to find a value $\text{S}\!\check{V}$ with:

[TABLE]

where $\text{S}\!V^{i}$ is the maximin value of player $i$ in the true game $M$. To do so, we return a lower bound value for the optimistic maximin policy $\hat{\pi}_{\text{S}\!V_{k}}^{i}$ of player $i$. We begin by computing in polynomial time444For example by using linear programming [10, 1]. the (stationary) maximin policy for the game $\hat{M}$ with largest rewards. We then compute the (deterministic, stationary) best response policy $\check{\pi}_{\text{S}\!\hat{V}}^{-i}$ using the game $\check{M}$ with the lowest rewards. The detailed steps are available in Algorithm 1. This results in a lower bound on the maximin value satisfying (3) as proven in Lemma 1.

§4 Computing an EBS policy.

Armed with the optimistic game and the optimistic maximin value, we can now easily compute the corresponding optimistic advantage game whose rewards are denoted by $\hat{r}_{+}$. An EBS policy $\tilde{\pi}_{k,\text{Eg}}$ is computed using this advantage game. The key insight to do so is that the EBS involves playing a single deterministic stationary policy or combine two deterministic stationary policies (Proposition 1). Given that the number of actions is finite we can then just loop through each pairs of joint-actions and check which one gives the best EBS score. The score (justified in the proof of Proposition 2 in Appendix C.4.) to use for any two joint-actions $a$ and $a^{\prime}$ is: $\text{score}(a,a^{\prime})=\min_{i\in\{1,2\}}w(a,a^{\prime})\cdot\hat{r}^{i}_{+}(a)+(1-w(a,a^{\prime}))\cdot r_{+}^{i}(a^{\prime})$ with $w$ as follows:

[TABLE]

And the policy $\tilde{\pi}_{k,\text{Eg}}$ is such that

[TABLE]

§5 Policy Execution

We always play the optimistic EBS policy $\tilde{\pi}_{k,\text{Eg}}$ unless one of the following three events happens:

• •

The probable error on the maximin value of one player is too large. Indeed, the error on the maximin value can become too large if the weighted bound on the actions played by the maximin policies is too large. In that case, we play the action causing the largest error.

• •

The small $\epsilon$-error large regret issue is probable: Proposition 2 implies that the small $\epsilon$-error large regret issue may only happen if the player with the lowest ideal advantage value (the maximum advantage under the condition that the advantage of the other player is non-negative) is receiving it when playing an EBS policy. This allows Algorithm 2 to check for this player and plays the action corresponding to its ideal advantage as far as the other player is still receiving $\epsilon_{k}$-close to its EBS value (Line 5 to 15 in Algorithm 2).

• •

The probable error on the EBS value of one player is too large This only happens if we keep not playing the EBS policy due to the small $\epsilon$-error large regret issue. In that case, the error on the EBS value used to detect the small $\epsilon$-error large regret issue might become too large making the check for the small $\epsilon$-error large regret issue irrelevant. In that case, we play the action of the EBS policy responsible for the largest error.

4 Theoretical analysis

Before we present theoretical analysis for the learning algorithm, we discuss the existence and uniqueness of the EBS value, as well as the type of policies that can achieve it.

Properties of the EBS

Fact 1 allows us to restrict our attention to stationary policies since it means that any (optimal) value achievable can be achieved by a stationary (correlated-) policy and Fact 2 means that the egalitarian always exists and is unique providing us with a good benchmark to compare against. Fact 1 and 2 are resp. justified in Appendix C.1 and C.2.

Fact 1** (Achievable values for both players).**

Any achievable value $V=(V^{i},V^{-i})$ for the players can be achieved by a stationary correlated-policy.

Fact 2** (Existence and Uniqueness of the EBS value for stationary policies).**

If we are restricted to the set of stationary policies, then the EBS value defined in Definition 7 exists and is unique.

The following Proposition 1 strengthens the observation in Fact 1 and establishes that a weighted combination of at most two joint-actions can achieve the EBS value. This allows for an efficient algorithm that can just loop through all possible pairs of joint-actions and check for the best one. However, given any two joint-actions one still needs to know how to combine them to get an EBS value. This question is answered by proposition 2.

Proposition 1** (On the form of an EBS policy).**

Given any 2 player repeated game, the EBS value can always be achieved by a stationary policy with non-zero probability on at most two joint-actions.

Sketch.

We follow the same line of reasoning used in [18] by showing that the EBS value lies on the outer boundary of the convex hull introduced in the proof of Fact 1. This immediately implies the proposition. Details are available in Appendix C.3. ∎

Proposition 2** (Finding an EBS policy).**

Let us call the ideal advantage value $V^{i}_{+I}$ of a player $i$, the maximum advantage that this player can achieve under the restriction that the advantage value of the other player is non-negative. More formally: $V^{i}_{+I}=\max_{\pi\mid V^{-i}_{+}(\pi)\geq 0}V^{i}_{+}(\pi)$. The egalitarian advantage value for the two players is exactly the same unless there exists an EBS policy that is deterministic stationary where at least one player (necessarily including the player with the lowest ideal advantage value) is receiving its ideal advantage value.

Sketch.

Since there is an EBS policy playing only two joint-actions (by Proposition 1), we show how to optimally combine any two joint-actions. The proposition then follows directly. More details is available in Appendix C.4 ∎

Regret Analysis

The following theorem 1 gives us a high probability upper bound on the regret in self-play against the EBS value, a result achieved without the knowledge of $T$.

Theorem 1** (Individual Rational Regret for Algorithm 3 in self-play).**

After running Algorithm 3 in self play with $\delta_{t_{k}}=\frac{\delta}{k\cdot t_{k}}$ and $\epsilon_{t_{k}}=2\cdot\left(\frac{\lvert\mathcal{A}\rvert\ln t_{k}}{t_{k}}\right)^{1/3}$ for any rounds $T\geq\lvert\mathcal{A}\rvert$, with probability at least $1-\frac{\delta^{4}}{3T}$, the individual rational regret (definition 9 ) for each player is upper bounded as:

[TABLE]

Sketch.

The structure of the proof follows that of [16]. The key step is to prove that the value of policy $\tilde{\pi}_{k}$ returned by Algorithm 2 in our plausible set is $\epsilon$-close to the EBS value in the true model (optimism). In our case, we cannot always guarantee this optimism. Our proof identifies the concerned cases and show that they cannot happen too often (Lemma 4 in Appendix B.1). Then for the remaining cases, Lemma 3 shows that we can guarantee the optimism with an error of $4\epsilon_{t_{k}}$. The step-by-step detailed proof is available in Appendix B.1. ∎

By definition of EBS, Theorem 1 also applies to the safety regret. However in Theorem 2, we show that the optimistic maximin policy enjoys near-optimal safety regret of $\operatorname{\mathcal{O}}(\sqrt{T})$.

Theorem 2** (Safety Regret of policy $\tilde{\pi}_{\text{S}\!V_{k}}^{i}$ in Algorithm 1).**

Consider a safe algorithm for player $i$ obtained by playing the policy $\tilde{\pi}_{\text{S}\!V_{k}}^{i}$ with $\delta_{t_{k}}=\frac{\delta}{k\cdot t_{k}}$. After any rounds $T\geq\lvert\mathcal{A}\rvert$ against any opponent, then with probability at least $1-\frac{\delta^{4}}{3T}$, the safe regret (definition 8) of this policy is upper-bounded by:

[TABLE]

Sketch.

The proof works similarly to that of Theorem 1 by observing that here we can always guarantee optimism. A more detailed proof is available in Appendix D. ∎

Lower bounds for the individual rational regret

Here we establish a lower bound of $\Omega\left(T^{2/3}\right)$ for any algorithm trying to learn the EBS value. This shows that our upper bound is optimal up to logarithm-factors. The key idea in proving this lower bound is the example illustrated by Table 2. In that example, the rewards of the first player are all $\frac{1}{2}$ and the second player has an ideal value of $1$. However, 50% of the times a player cannot realize its ideal value due to an $\epsilon$-increase in a single joint-action for both players. The main intuition behind the proof of the lower bound is that any algorithm that wants to minimize regret can only try two things (a) detect whether there exists a joint-action with an $\epsilon$ or if all rewards of the first player are equal. (b) always ensure the ideal value of the second player. To achieve (a) any algorithm needs to play all joint-actions for $\frac{1}{\epsilon^{2}}$ times. Picking $\epsilon=T^{-1/3}$ ensures the desired lower bound. The same $\epsilon$ would also ensure the same lower bound for an algorithm targeting only (b). Appendix E formally proves this lower bound.

Theorem 3** (Lower bounds).**

*For any algorithm $\Lambda$, any natural numbers $|\mathcal{A}^{1}|\geq 2$, $|\mathcal{A}^{2}|\geq 2$, $T\geq|\mathcal{A}^{1}|\times|\mathcal{A}^{2}|$, there is a general sum game with $\lvert\mathcal{A}\rvert=|\mathcal{A}^{1}|\times|\mathcal{A}^{2}|$ joint-actions such that the expected individual rational regret of $\Lambda$ after $T$ steps is at least $\Omega\left(T^{2/3}\frac{\lvert\mathcal{A}\rvert^{1/3}}{4}\right)$.

5 Conclusion and Future Directions

In this paper, we illustrated a situation in which typical solutions for self-play in repeated games, such as single-stage equilibrium or sum of rewards, are not appropriate. We propose the usage of an egalitarian bargaining solution (EBS) which guarantees each player to receive no less than their maximin value. We analyze the properties of EBS for repeated games with stochastic rewards and derive an algorithm that achieves a near-optimal finite-time regret of $\operatorname{\mathcal{O}}(T^{2/3})$ with high probability. We are able to conclude that the proposed algorithm is near-optimal, since we prove a matching lower bound up to logarithmic-factor. Although our results imply a $\operatorname{\mathcal{O}}(T^{2/3})$ safety regret (i.e. compared to the maximin value), we also show that a component of our algorithm guarantees the near-optimal $\operatorname{\mathcal{O}}(\sqrt{T})$ safety regret against arbitrary opponents.

Our work illustrates an interesting property of the EBS which is: it can be achieved with sub-linear regret by two individually rational agents who are uncertain about their utility. We wonder if other solutions to the Bargaining Problem such as the Nash Bargaining Solution or the Kalai–Smorodinsky Solution also admit the same property. Since the EBS is an equilibrium, another intriguing question is whether one can design an algorithm that converges naturally to the EBS solution against some well-defined class of opponents.

Finally, a natural and interesting future direction for our work is its extension to stateful games such as Markov games.

Appendix A Notations and terminology

We will use action to mean joint-actions unless otherwise specified. We will denote the players as $i$ and $-i$. This is to be understood as follows: if there are two players $\{1,2\}$, when $i=1$, then $-i=2$ and when $i=2$, $-i=1$. The true but unknown game will be denoted as $M$ whereas the plausible set of games we consider at epoch $k$ will be denoted by $\mathcal{M}_{k}$. An EBS policy in the true game $M$ will be denoted by $\pi_{\scriptscriptstyle\text{Eg}}$ and its value by $V_{\scriptscriptstyle\text{Eg}}$. If for the EBS value in $M$, the player with the lowest ideal advantage value is receiving it, we will denote this player by $p^{-}$ while the other player will be $p^{+}$. The EBS policy in this situation will be denoted as $a^{*}$ (it is guaranteed to be a single joint-action).

$\bar{r}$ will be used to denote empirical mean rewards and in general $\;\bar{}\;$ is used to mean a value computed using empirical $\bar{r}$. $\tilde{r}$ will be used to mean the rewards from the upper limit game in our plausible set, while $\hat{r}$ will be used to mean the rewards from the lower limit game in our plausible set. Also, in general $\;\tilde{}\;$ while be used to mean a value computed using $\tilde{r}$ and $\;\hat{}\;$ to mean a value computed using $\hat{r}$.

$k$ will be used to denote the current epoch. $N_{k}(a)$ the number of rounds action $a$ has been played in epoch $k$ — $N_{k}$ the number of rounds epoch $k$ has lasted — $t_{k}$ the number of rounds played up to epoch $k$ — $N_{t_{k}}(a)$ the number of rounds action $a$ has been played up to round $t_{k}$ — $\bar{r}_{t}^{i}(a)$ the empirical average rewards of player $i$ for action $a$ at round $t$. $m$ will be used to denote the total number of epochs up to round $T$.

Appendix B Proof of Theorem 1

Theorem** (1).**

After running Algorithm 3 in self play with $\delta_{t_{k}}=\frac{\delta}{k\cdot t_{k}}$ and $\epsilon_{t_{k}}=2\cdot\left(\frac{\lvert\mathcal{A}\rvert\ln t_{k}}{t_{k}}\right)^{1/3}$ for any rounds $T\geq\lvert\mathcal{A}\rvert$, with probability at least $1-\frac{\delta^{4}}{3T}$, the individual rational regret (definition 9 ) for each player is upper bounded as:

[TABLE]

B.1 Regret analysis for the egalitarian algorithm in self-play

The proof is similar to that of UCRL2 [16] and KL-UCRL [12]. As the algorithm is divided into epochs, we first show that the regret bound within an epoch is sub-linear. We then combine those per-epoch regret terms to get a regret for the whole horizon simultaneously. Both of these regrets are computed with the assumption that the true game $M$ is within our plausible set. We then conclude by showing that this is indeed true with high probability. Let’s first start by decomposing the regret.

Regret decomposition

Here we decompose the regret in each round $k$. We start by defining the following event $E$,

[TABLE]

[TABLE]

We have:

[TABLE]

In the following, we will use Hoeffding’s inequality to bound the last term of Equation 13, similarly to Section 4.1 in [16]. In particular, with probability at least $1-\delta^{\prime}$:

[TABLE]

where $\Delta_{k}$ is the regret per-epoch defined by

[TABLE]

Regret when the event E defined by (6) is False and the true Model is in our plausible set

We will now simplify the notation by using $\Delta_{k,\neg E}$ to mean that the expression is condition on $E$ being False. We can thus bound $\Delta_{k,\neg E}$:

[TABLE]

where Equation (16) comes from the fact that when $E=0$, $\hat{V}_{k}^{i}\geq V_{\scriptscriptstyle\text{BR}}^{i}-4\epsilon_{t_{k}}$ (See Lemma 3). Equation (17) comes from the fact that we assume $M\in\mathcal{M}_{k}$ meaning $\lvert\bar{r}^{i}(a)-\operatorname{\mathbb{E}}r^{i}(a)\rvert\leq\frac{C_{r}(t_{k})}{\sqrt{N_{t_{k}}(a)}}$. Equation (18) comes from the fact the egalitarian solution involves playing one joint-action with probability $w_{k}\in[0,1]$ and another joint-action with probability $1-w_{k}$; since it is always possible to bound $w_{k}$ as $\frac{n}{N_{k}}\leq w_{k}\leq\frac{n+1}{N_{k}}$ with $n\in\mathbb{N}$ a non-negative integer, and by construction the players play as close as possible to $w_{k}$, then the error is bounded by $\frac{1}{N_{k}}\leq\frac{1}{N_{k}(a)}$.

We are now ready to sum up the per-epoch regret over all epochs for which the event $E$ is false. We have:

[TABLE]

Now assuming $\epsilon_{t_{k}}=C_{e}\cdot\left(\frac{\lvert\mathcal{A}\rvert\ln t_{k}}{t_{k}}\right)^{1/3}$, we have:

[TABLE]

Using Appendix C.3 in [16], we can conclude that

[TABLE]

Similarly [16] Equation (20) shows that:

[TABLE]

Furthermore [16] shows that:

[TABLE]

Combining all the above results lead to

[TABLE]

Combining with probability of failure

Combining (23) with Lemma 4 bounding the number of times event $E$ is true, together with Proposition 3 justifying the high probability from $t\geq\max\{3,\left(T\lvert\mathcal{A}\rvert\right)^{1/4}\}$, noticing that up to $t=\max\{3,\left(T\lvert\mathcal{A}\rvert\right)^{1/4}\}$ our maximum regret is $\max\{6,2\left(T\lvert\mathcal{A}\rvert\right)^{1/4}\}$, picking $\delta^{\prime}$ in (14) as $\delta^{\prime}=\frac{\delta^{4}}{T}\cdot\left(\frac{99.375-\pi^{4}}{45}\right)$ and using $\max_{k}C_{r}(t_{k})=C_{r}(t_{m})$ leads to:

[TABLE]

with a failure probability of $\frac{\delta^{4}}{3T}$

Picking $C_{e}=2$ leads to the statement of Theorem 1.

Proposition 3** (Probability of Failure).**

If Algorithm 3 is run with the plausible set constructed with $\delta_{t_{k}}=\frac{\delta}{k\cdot t_{k}}$, then the probability of failure from round $t=\max\{3,\left(T\lvert\mathcal{A}\rvert\right)^{1/4}\}$ to round $T$ with $T\geq 3$ is upper bounded as:

[TABLE]

Proof.

Using Lemma 5, we know that at any epoch $k$, our plausible set contains the true model with probability at least $1-2\lvert\mathcal{A}\rvert\delta_{t_{k}}^{4}$, meaning that the probability of failure is $2\lvert\mathcal{A}\rvert\delta_{t_{k}}^{4}$. We want to combine these failure probabilities for all possible epochs (after round $\max\{3,\left(T\lvert\mathcal{A}\rvert\right)^{1/4}\}$). For that, consider $\delta_{t_{k}}=\frac{\delta}{\cdot k\cdot t_{k}}$. Let $k_{t}$ the smallest round such that $t_{k_{t}}\geq\max\{3,\left(T\lvert\mathcal{A}\rvert\right)^{1/4}\}$. Using a union bound (Fact 4) we have that the probability $\operatorname{\mathbb{P}}\left\{\bigcup_{k\mid t_{k}\geq\max\{3,\left(T\lvert\mathcal{A}\rvert\right)^{1/4}\}}M\notin\mathcal{M}_{k}\right\}$ up to round $T$ is:

[TABLE]

(28) comes from $\sum_{k=1}^{\infty}\frac{1}{k^{4}}=\frac{\pi^{4}}{90}$ and $k_{t}\geq 3$ when $t_{k_{t}}\geq 3$ due to the doubling trick used in Algorithm 3.

∎

The following fact comes directly by construction of the Algorithm.

Fact 3** (Error on the maximin policy).**

For any player $i$ and epoch $k$ for which the true model $M$ is within our plausible set $\mathcal{M}_{k}$ and Event $E_{1}$ defined by (7) is False, then:

[TABLE]

Lemma 1** (Pessimism and Optimism of the maximin value).**

For any player $i$ and epoch $k$ for which the true model $M$ is within our plausible set $\mathcal{M}_{k}$, the maximin value computed satisfies:

[TABLE]

Proof.

Pessimism of the maximin value. Let $\hat{V}$ denote the lower bound on the value through the reward estimate $\bar{r}-C$ and $\tilde{V}$ the corresponding upper bound through $\bar{r}+C$. By definition,

[TABLE]

As a result we have:

[TABLE]

Optimism of the maximin value

We have:

[TABLE]

∎

Lemma 2** (Optimism of the advantage game).**

This lemma prove that the advantage value for any policy $\pi$ in our optimistic model is greater than in the true model. For any policy $\pi$, player $i$ and epoch $k$ for which the true model $M$ is within our plausible set $\mathcal{M}_{k}$:

[TABLE]

Proof.

First, we prove that the advantage value for any policy $\pi$ in our optimistic model is greater than in the true model. We have:

[TABLE]

where (48) comes from Lemma 1. ∎

Lemma 3** (Optimism of the Policy computation).**

For any epoch $k$ for which the true model $M$ is in our plausible set $\mathcal{M}_{k}$ and Event $E$ defined by (6) is False, then for any player $i$, we have:

[TABLE]

Proof.

Immediate by combining Lemma (10) (11), (12) ∎

Lemma 4** (Number of times event $E$ defined by (6) is True).**

After any number $T$ of rounds for which the true model $M$ was in our plausible set $\mathcal{M}_{k}$ at every epochs $k$ up to $T$, the number $N_{T}(E=\text{True})$ of rounds for which Event $E$ defined by (6) is True satisfy:

[TABLE]

Proof.

The proof is an immediate consequence of combining Lemmas 6,7,8, 9. ∎

Lemma 5** (High probability of the plausible set).**

At any epoch $k$, with probability at least $1-2\lvert\mathcal{A}\rvert\delta^{4}$, the true model $M$ is in our plausible set constructed using (1).

Proof.

This is a direct application of Hoeffding’s bound (Lemma 14) on each action individually and then using a union bound (Fact 4) over all actions. ∎

Lemma 6** (Number of times an action from the maximin policy is played).**

After any number $T$ of rounds for which the true model $M$ was in our plausible set $\mathcal{M}_{k}$ at every epochs $k$ up to $T$, the number $N_{T}(E_{1}=\text{True})$ of rounds for which event $E_{1}$ defined by (7) is true satisfy:

[TABLE]

Proof.

By definition, the event $E_{1}$ is true when $\tilde{\pi}_{k}=\tilde{a}_{\text{S}\!V_{k}}$. We have:

[TABLE]

Summing over all actions, using the fact that during epoch $k$ no action is played more than twice the number of times it was played at the beginning of epoch $k$ and replacing $\epsilon_{t_{k}}$ by its value, leads to the statement of the lemma. ∎

Lemma 7** (Number of times event defined by (8) is True).**

After any number $T$ of rounds for which the true model $M$ was in our plausible set $\mathcal{M}_{k}$ at every epochs $k$ up to $T$, the number $N_{T}(E_{2}=\text{True})$ of rounds for which event $E_{2}$ defined by (8) is true satisfy:

[TABLE]

Proof.

By definition, the event is true when $\tilde{\pi}_{k}=\tilde{a}_{k,\text{Eg}}$. We have:

[TABLE]

And the remainder of the proof follows the proof of Lemma 6 ∎

Lemma 8** (Number of times Event defined by (9) is True).**

After any number $T$ of rounds for which the true model $M$ was in our plausible set $\mathcal{M}_{k}$ at every epochs $k$ up to $T$, Event $E_{3}$ defined by (9) is always False.

Proof.

[TABLE]

(54) comes from $a_{*}\notin\tilde{\mathcal{A}}_{p^{-}}$ (by definition)

(56) comes from (since the true model is in our plausible set)

[TABLE]

(57) comes from $\operatorname{\mathbb{E}}r_{+}^{p^{-}}(a_{\text{Eg}})\geq V_{+}^{p^{-}}(\tilde{\pi}_{k,\text{Eg}})$ (since by assumption $\pi_{Eg}=a_{*}$) and $2C(\tilde{\pi}_{k,\text{Eg}})\leq\epsilon_{t_{k}}$ (by assumption). ∎

Lemma 9** (Number of times Event defined by (10) is True).**

After any number $T$ of rounds for which the true model $M$ was in our plausible set $\mathcal{M}_{k}$ at every epochs $k$ up to $T$, the number $N_{T}(E_{4}=\text{True})$ of rounds for which Event $E_{4}$ defined by (10) is True satisfy:

[TABLE]

Proof.

The condition $E_{4}$ is: $a_{*}\in\tilde{\mathcal{A}}_{p^{-}}\land\tilde{\pi}_{k}=\tilde{a}_{p^{-}}\land p^{+}\in\tilde{\mathcal{P}}\mid\pi_{Eg}=a_{*},V_{+}^{p^{+}}(a_{*})>V_{+}^{p^{-}}(a_{*})+2\epsilon_{t_{k}}$

[TABLE]

where (58) comes (by definition of event $E_{4}$) $\tilde{\pi}_{k}=\tilde{a}_{p^{-}}\land p^{+}\in\tilde{\mathcal{P}}$

(59) comes from (by definition of event $E_{4}$) $a_{*}\in\tilde{\mathcal{A}}_{p^{-}}$ and the definition of $\tilde{a}_{p^{+}}$

(60) comes from using Lemma 2 since the true model is in our plausible set.

(61) comes from the definition of event $E_{4}$ that $V_{+}^{p^{+}}(a_{*})>V_{+}^{p^{-}}(a_{*})+2\epsilon_{t_{k}}$

(62) comes from (since the true model is in our plausible set)

[TABLE]

(63) comes from Lemma 1 (since the true model is in our plausible set) and Fact 3 (since Event $E_{1}$ is False)

(64) comes from $\operatorname{\mathbb{E}}r_{+}^{p^{-}}(a_{*})\geq\operatorname{\mathbb{E}}r^{p^{-}}_{+}(\tilde{a}_{p^{-}})$ since $\pi_{\text{Eg}}=a_{*}$

Summing over all actions, using the fact that during epoch $k$ no action is played more than twice the number of times it was played at the beginning of epoch $k$ and replacing $\epsilon_{t_{k}}$ by its value, leads to the statement of the lemma. ∎

Lemma 10** (Optimism of the Policy computation when egalitarian advantage is identical).**

For any epoch $k$ for which the true model $M$ is in our plausible set $\mathcal{M}_{k}$, the two players have the same egalitarian advantage value in $M$ and Event $E$ defined by (6) is False, then for any player $i$, we have:

[TABLE]

Proof.

We have:

[TABLE]

(69) comes by definition of $\tilde{\pi}_{k,\text{Eg}}$ (i.e the policy maximizing the minimum advantage).

(71) comes by assumption that in the true model both players have the same egalitarian advantage.

(74) comes from Lemma 1 and (75) from Fact 3. ∎

Lemma 11** (Optimism of the Policy computation for the player $p^{-}$).**

For any epoch $k$ for which Event $E$ defined by (6) is False, the true model $M$ is in our plausible set $\mathcal{M}_{k}$, one player (named $p^{-}$) is receiving its maximum egalitarian advantage value in $M$, then for player $p^{-}$, we have:

[TABLE]

Proof.

The proof is similar to the one of Lemma 10 but using the fact that here, only $p^{-}$ is guaranteed to have the minimum egalitarian advantage in the true model. We have:

[TABLE]

∎

Lemma 12** (Optimism of the Policy computation for player $p^{+}$).**

For any epoch $k$ for which the true model $M$ is in our plausible set $\mathcal{M}_{k}$, one player (named $p^{-}$, the other is named $p^{+}$) is receiving its maximum egalitarian advantage value in $M$ and Event $E$ defined by (6) is False, then for player $p^{+}$, we have:

[TABLE]

Proof.

First, we decompose the different cases that appear when Event $E$ (6) is False. Then, we will treat each cases separately.

We have:

[TABLE]

(88) comes from the assumption that $\pi_{Eg}=a_{*}$ since $p^{-}$ is receiving its maximum egalitarian advantage in $M$

(89) comes from the fact that by construction of the Algorithm it is impossible for $\tilde{\pi}_{k}\neq\tilde{a}_{k,\text{Eg}}$ and $2C_{t_{k}}(\tilde{\pi}_{k,\text{Eg}})>\epsilon_{t_{k}}$ to be True simultaneously.

(90) comes from the fact that is it not possible to have $a_{*}\in\tilde{\mathcal{A}}_{p^{-}}$ and $a_{*}\notin\tilde{\mathcal{A}}_{p^{-}}$ simultaneously.

Now let’s decompose $\tilde{\pi}_{k}\neq\tilde{a}_{p^{-}}\lor p^{+}\notin\tilde{\mathcal{P}}$ part of (90)

[TABLE]

Replacing (92) into (90) gives us:

[TABLE]

We will now bound the EBS value of $p^{+}$ for each term of (93) as a separate cases. .

Case: $\tilde{\pi}_{k}\neq\tilde{a}_{\text{S}\!V_{k}}\land\tilde{\pi}_{k}\neq\tilde{a}_{k,\text{Eg}}\land a_{*}\in\tilde{\mathcal{A}}_{p^{-}}\land V_{+}^{p^{+}}(a_{*})\leq V_{+}^{p^{-}}(a_{*})+2\epsilon_{t_{k}}$

We have:

[TABLE]

Case: $\tilde{\pi}_{k}\neq\tilde{a}_{\text{S}\!V_{k}}\land\tilde{\pi}_{k}\neq\tilde{a}_{k,\text{Eg}}\land a_{*}\in\tilde{\mathcal{A}}_{p^{-}}\land\tilde{\pi}_{k}\neq\tilde{a}_{p^{-}}\land p^{+}\in\tilde{\mathcal{P}}$

[TABLE]

Case: $\tilde{\pi}_{k}\neq\tilde{a}_{\text{S}\!V_{k}}\land\tilde{\pi}_{k}\neq\tilde{a}_{k,\text{Eg}}\land a_{*}\in\tilde{\mathcal{A}}_{p^{-}}\land p^{+}\notin\tilde{\mathcal{P}}$

[TABLE]

(115) is due to the fact that since $p^{+}\notin\mathcal{P}$ we have: $\max_{a\in\tilde{\mathcal{A}}_{p^{-}}}\tilde{V}^{p^{+}}_{+}(a)\leq\hat{V}_{+}^{p^{+}}(\tilde{\pi}_{k,\text{Eg}})$

Combining (104), (111), (119) leads to the statement of the Lemma.

∎

Appendix C On the Egalitarian Bargaining solution

C.1 Achievable values for both players

Fact** (1).**

Any achievable value $V=(V^{i},V^{-i})$ for the players can be achieved by a stationary correlated-policy.

Proof.

We start by showing that values for deterministic stationary policies exists and are unique, then we conclude by showing that any values can be achieved simultaneously for both player by a single stationary correlated-policy.

For any player $i$, the value as defined by Definition 4 of any stationary policy exists and is unique [25] since the game would be equivalent to a $1$-state Markov Decision Process. As a result, the value of deterministic stationary policies (i.e joint-actions) exists.

When player $i$ play with a deterministic stationary policy $a^{i}$ and player $-i$ plays with a deterministic stationary policy $a^{-i}$, the values for the two players can be visualized as a point $x=\left(V^{i}(a^{i},a^{-i}),V^{-i}(a^{i},a^{-i})\right)=(x^{i},x^{-i})$ in a two-dimensional space.

Following [21], we consider the set of all pairs of (values for) deterministic policies $X=\{\left(V^{i}(a^{i},a^{-i}),V^{-i}(a^{i},a^{-i})\right)\;\forall a^{i}\in\mathcal{A}^{i},a^{-i}\in\mathcal{A}^{-i}\}$ for the two players. All the points $x\in X$ can be achieved as value for the two players in the repeated game, simply by repeatedly playing the corresponding joint-action.

Consider the convex hull $\mathcal{C}$ of the set of points $x\in X$. This means that any point in the convex hull can be expressed as a weighted linear combination of the points $x\in X$ where the weights sum up to 1. Those weights can thus be seen as probabilities which allows us to affirm that any point in the convex hull can be achieved as values for the two players in the repeated game (by playing the corresponding stationary policy with the weight as probabilities). On the other side, any achievable values for the two players belongs to the convex since hull which follows from the definition of convex hull. In conclusion, the convex hull represents exactly the set of all achievable values for the two players. And since any point in the convex hull is achievable by a stationary policy, this concludes our proof. ∎

C.2 Existence and Uniqueness of the EBS value for stationary policies

Fact** (2).**

If we are restricted to the set of stationary policies, then the EBS value defined in Definition 7 exists and is unique.

Proof.

[15] proves that the EBS value as defined in Definition 7 always exists and is unique for any bargaining problem that is convex, closed, of non-empty Pareto frontier and non degenerate (i.e there exists a point greater or equal than the disagreement point).

To conclude the proof of this fact, it is then enough to prove that we have a bargaining problem satisfying those properties. We consider the bargaining problem induced by the repeated game.

Here our disagreement point is the maximin value.

From the proof of Fact 1, we can see that the repeated game with stochastic rewards can be replaced by another one with deterministic rewards corresponding to the values of joint-actions. As a result, the maximin value exists and is unique. Also, there always exists a unique (one-stage) Nash Equilibrium which is greater in value than the maximin value of both players [20]. This means that player can always get their maximin value. So our bargaining problem is non-degenerate.

Finally, using the same convex hull as in the proof of Fact 1, we can see that the set of achievable values is convex. This set is also closed since the joint-actions are finite and the rewards are bounded. And there always exists one Pareto efficient policy (Any policy achieving the maximum value for one player). This concludes the proof. ∎

C.3 On the form of an EBS policy

Proposition** (1).**

Given any 2 player repeated game, the EBS value can always be achieved by a stationary policy with non-zero probability on at most two joint-actions.

Proof.

Let’s recall that the EBS value maximize the minimum possible for any player and as a result if we have a value where the minimum advantage can’t be improved anymore, then we have the EBS value provided that we also maximize the value of the second player if possible.

Now let’s consider the convex hull defined in the proof of Fact 1. The egalitarian point will be found on the outer boundary of the convex hull – the minimum value of any internal point can be increased by moving to a point above it and to the right and higher minimum means higher EBS value. This implies that the egalitarian point can be expressed by a weight vector $w$ that has non-zero weight on only one or two $x\in X$, since the convex hull is a two-dimensional polygonal region bounded by line segments.

∎

C.4 Finding an EBS policy

Proposition** (2).**

Let us call the ideal advantage value $V^{i}_{+I}$ of a player $i$, the maximum advantage that this player can achieve under the restriction that the advantage value of the other player is non-negative. More formally: $V^{i}_{+I}=\max_{\pi\mid V^{-i}_{+}(\pi)\geq 0}V^{i}_{+}(\pi)$. The egalitarian advantage value for the two players is exactly the same unless there exists an EBS policy that is deterministic stationary where at least one player (necessarily including the player with the lowest ideal advantage value) is receiving its ideal advantage value.

Proof.

From proposition 1 we can achieve the EBS value by combining at most two deterministic stationary policies. We will prove this proposition (2) for any two possible deterministic stationary policies (by considering a repeated game with only the corresponding joint-actions available), which immediately means that the proposition 2 is also true for the EBS value in the full repeated game.

Consider any two deterministic stationary policy of advantage values ($(x^{1}_{1},x^{2}_{1}),(x^{1}_{2},x^{2}_{2})$). We will now show how to compute the weight $w=\operatorname*{argmax}_{w}\min_{i\in\{1,2\}}w*x^{i}_{1}+(1-w)x^{i}_{2}$.

Case 1:

$x^{1}_{1}\leq x^{2}_{1}$ and $x^{1}_{2}\leq x^{2}_{2}$. This basically means that the advantage value of player $2$ is always higher or equal than that of the player $1$. So the minimum is maximized by playing the policy maximizing the value of player $1$. So, $w=0$ and we have a single deterministic stationary policy where the player with the lowest ideal advantage receives it.

Case 2:

$x^{1}_{1}\geq x^{2}_{1}$ and $x^{1}_{2}\geq x^{2}_{2}$. This is essentially Case 1 with the role of player $1$ and $2$ exchanged. Here $w=1$.

If both Case 1 and Case 2 do not hold, it means that for the first policy, one player receives an advantage value strictly greater than that of the other player while the situation is reversed for the second policy. Without loss of generality we can assume this player is $1$ (if this is not the case, we can simply switch the id of the policy) which leads to Case 3.

Case 3:

$x^{1}_{1}>x^{2}_{1}$ and $x^{1}_{2}<x^{2}_{2}$ In this case, the optimal $w$ is such that $w=\frac{x^{2}_{2}-x^{1}_{2}}{(x^{1}_{1}-x^{1}_{2})+(x^{2}_{2}-x^{2}_{1})}$. This weight $w$ is clearly between the open interval $]0,1[$. This means that we have exactly two distinct policies. Plugging in the weight shows that the advantage value of both player is the same, which completes the proof. ∎

Appendix D Regret analysis for the safe policy against arbitrary opponents

Theorem** (2).**

Consider a safe algorithm for player $i$ obtained by playing the policy $\tilde{\pi}_{\text{S}\!V_{k}}^{i}$ with $\delta_{t_{k}}=\frac{\delta}{k\cdot t_{k}}$. After any rounds $T\geq\lvert\mathcal{A}\rvert$ against any opponent, then with probability at least $1-\frac{\delta^{4}}{3T}$, the safe regret (definition 8) of this policy is upper-bounded by:

[TABLE]

Proof.

The proof is similar to the one for Theorem 1. However, here we don’t have to deal with the event $E$ defined in (6) which is thus taken to always be False.

Also, we are always optimistic (against the true maximin value when the true model $M$ is within our plausible set $\mathcal{M}_{k}$) by playing policy $\tilde{\pi}_{\text{S}\!V}^{i}$ computed in (2). Indeed, for any opponent policy $\pi^{-i}$,

[TABLE]

Note that this is not a contradiction to the upper bound in Lemma 1 since the $\text{S}\!\hat{V}^{i}$ mentioned in Lemma 1 is computed using the value $\hat{V}$ (rather than $\tilde{V}$) lower than the empirical values as shown by (4).

As a result, the corresponding $\epsilon_{t_{k}}$ used by step (16) in the proof of Theorem 1 is 0.

Finally, the arguments justifying step (18) in the proof of Theorem 1 does not hold anymore since we are now playing a completely random policy. Instead, we can bound the deviation of $\sum_{k=1}^{m}\sum_{a\in\mathcal{A}}N_{k}(a)\left(\hat{V}_{k}^{i}-\hat{r}^{i}(a)\right)$ using Chernoff-bound. Combining those remarks into the proof of Theorem 1 leads to statement of Theorem 2. ∎

Appendix E Proof of the Lower bounds in Theorem 3

Theorem** (3).**

*For any algorithm $\Lambda$, any natural numbers $|\mathcal{A}^{1}|\geq 2$, $|\mathcal{A}^{2}|\geq 2$, $T\geq|\mathcal{A}^{1}|\times|\mathcal{A}^{2}|$, there is a general sum game with $\lvert\mathcal{A}\rvert=|\mathcal{A}^{1}|\times|\mathcal{A}^{2}|$ joint-actions such that the expected individual rational regret of $\Lambda$ after $T$ steps is at least $\Omega\left(T^{2/3}\frac{\lvert\mathcal{A}\rvert^{1/3}}{4}\right)$.

Proof.

The proof is inspired by the one for bandits in Theorem 6.11 of [6]. First, we prove the theorem for deterministic stationary algorithms. The general case then follows by a simple argument. The main idea of the proof is to show that there exists a repeated game such that for any algorithm the expected regret is large (where the expectation is understood with respect to the random rewards from the repeated game).

Let the repeated games be as follows: The first player has $\lvert\mathcal{A}^{1}\rvert$ actions while the second player has $\lvert\mathcal{A}^{2}\rvert$ actions. At each round, the players’ rewards are generated independently from a Bernoulli distribution whose expectation depends on the joint-action as follows: For one action pair $a_{*}=(a^{1}_{i},a^{2}_{j})$, the rewards of the first player has expectation $\frac{1}{2}$ while the second player receives $1$. All the others joint-actions yield rewards with expectation $\frac{1}{2}$ to both players. However, we also selects uniformly at random one joint-action $a$ from the set of all joint-actions expect $a_{*}$ and with probability $\frac{1}{2}$, we switch the expectation of that action from $\frac{1}{2}$ to $\frac{1}{2}+\epsilon$ for both players. This selection can be thought as having a random variable $Z$ that selects $a_{*}$ with probability $\frac{1}{2}$ and, the remaining joint-actions with probability $\frac{1}{2(\lvert\mathcal{A}\rvert-1)}$

Let’s denote $\boldsymbol{r}^{T}$ the sequence of all generated rewards and $\boldsymbol{\mathcal{R}}^{T}$, the set of all possible sequence up to round $T$ Also, let $G^{i}_{.,T}$ be the sum of rewards obtained by an algorithm for player $i$ up to round $T$. Then for any (non-randomized) algorithm we have:

[TABLE]

where the expectation on the right-hand side is with respect to the random variables $r_{t}(a)$. Thus it suffices to bound, from above, the expected regret for the randomly chosen rewards.

First Observe that for player $i$:

[TABLE]

Computing the optimal EBS values for each player gives:

$V_{\scriptscriptstyle\text{Eg},Z\neq a_{*}}^{1}=\frac{1}{2}+\epsilon$, $V_{\scriptscriptstyle\text{Eg},Z\neq a_{*}}^{2}=\frac{1}{2}+\epsilon$, $V_{\scriptscriptstyle\text{Eg},Z=a_{*}}^{1}=\frac{1}{2}$,$V_{\scriptscriptstyle\text{Eg},Z=a_{*}}^{2}=1$

As a result, we can conclude about the optimal Egalitarian gain for both agents as:

[TABLE]

Now, we need to bound, from above the expected cumulative reward $\operatorname{\mathbb{E}}G_{\scriptscriptstyle T,\Lambda}^{i}$ for an arbitrary algorithm. To this end, fix a (deterministic stationary) algorithm and let $I_{t}$ denote the joint-action it chooses at round $t$. Clearly, $I_{t}$ is determined by the sequence of rewards $\boldsymbol{r}=\{r_{I_{1},1},\ldots r_{I_{t-1},t-1}\}$. Also, let $T_{a}=\sum_{t=1}^{T}\operatorname{\mathbb{I}}_{I_{t}=a}$ be the number of times action $a$ is played by the algorithm. Then, writing $\operatorname{\mathbb{E}}_{a}\text{for}\operatorname{\mathbb{E}}[.|Z=a]$, we may write

[TABLE]

where Line (131) follows directly from [6].

So now we have

[TABLE]

Next, we apply Lemma 13 to $T_{a}$ which is a function of the rewards sequence $\boldsymbol{r}$ since the actions of the algorithm $\Lambda$ are determined by the past rewards. Since $T_{a}\in[0,T]$

[TABLE]

Now letting $x=\sum_{a\in\mathcal{A}\backslash a_{*}}\operatorname{\mathbb{E}}_{a*}[T_{a}]$ and using the fact that $\sum_{a\in\mathcal{A}\backslash a_{*}}\sqrt{x}\leq\sqrt{x(A-1)}$, we have:

[TABLE]

Combining (135) with (133) we can bound the gain of the first agent as:

[TABLE]

Similarly, combining (135) with (134) and the fact that $T_{a_{*}}+\sum_{a\in\mathcal{A}\backslash a_{*}}T_{a}=T$, we can bound the gain of the second agent as:

[TABLE]

We can now derive the lower bound for regret of the first agent by combining (136), (124), (128)

[TABLE]

Similarly for the second agent by combining (137), (124), (129)

[TABLE]

We can now derive the overall regret of the algorithm (equal to the maximum of the regret of both agents) by looking for the $x$ leading to the smallest maximum. Picking $\epsilon=\min\{A^{1/3}T^{-1/3},\frac{\sqrt{0.43}}{2}\}$ and using the fact that $-\ln(1-z)\leq z+\frac{z^{2}}{2}+\frac{z^{3}}{2}$ for $z\in[0,0.43]$ give the lower bound of $\Omega\left(\frac{T^{2/3}A^{1/3}}{4}\right)$

This concludes the proof for deterministic stationary algorithms. The extension to randomized algorithms follows the same argument as in [6]. ∎

Lemma 13**.**

Let $f:\{(0,0),(0,1),(1,0),(1,1)\}^{T}\to[0,M]$ be any function defined on rewards sequences $\boldsymbol{r}^{T}$. Then for any action $a\in\mathcal{A}\backslash a_{*}$,

[TABLE]

Proof.

Similarly to Lemma A.1 in [2] we have

[TABLE]

Computing the KL-divergence similarly to Lemma A.1 in [2] leads to:

[TABLE]

The lemma follows by combining (140) and (142)

∎

Appendix F Previously Known results

Lemma 14** (Chernoff-Hoeffding bound [14]).**

Let $X_{1},X_{2}\ldots X_{n}$ be random variables with common range $[0,1]$ and such that $\operatorname{\mathbb{E}}\left[X_{t}\mid X_{1},\ldots X_{t-1}\right]=\mu$. Then for all $\epsilon\geq 0$

[TABLE]

Fact 4** (Union Bound also known as Boole’s inequality).**

For a countable set of events $A_{1},A_{2},\ldots$ we have:

[TABLE]

Appendix G Algorithms

G.1 Finding an EBS policy for a game with known rewards distribution

For any game $M$ with rewards $r$ this can be done using (5) with $\hat{r}_{+}$ replaced by $r_{+}$

G.2 Communication protocol

It is also important for the players to communicate since the policies of the players might need to be correlated to play the same joint-action. This communication555It is possible to remove this assumption and uses more cryptographically robust synchronization protocol with minimal communication. However, this is out of scope of this paper. is done through lexicographical ordering the policies using the unique actions identifier and player identifier assumed to be shared before the game start by both players. When a policy involve playing multiple actions with different probabilities, players simply play actions such that their empirical probability of play is as close as possible to the true policy probability. This is explained more formally in the function $\textsc{Play}($) of Algorithm 3.