Playing Markov Games Without Observing Payoffs

TL;DR

This paper extends the ability to learn and compete in symmetric Markov games without payoff observations, using only opponent actions and known transition dynamics, by reducing the problem to online learning.

Contribution

It introduces a formal framework for zero-sum symmetric Markov games with limited information and provides algorithms that achieve asymptotic performance guarantees.

Findings

Players can compete without payoff observations using only opponent actions.

Algorithms run in polynomial time relative to game size and episodes.

The approach applies to both matrix and Markov games, broadening robust learning scenarios.

Abstract

Optimization under uncertainty is a fundamental problem in learning and decision-making, particularly in multi-agent systems. Previously, Feldman, Kalai, and Tennenholtz [2010] demonstrated the ability to efficiently compete in repeated symmetric two-player matrix games without observing payoffs, as long as the opponents actions are observed. In this paper, we introduce and formalize a new class of zero-sum symmetric Markov games, which extends the notion of symmetry from matrix games to the Markovian setting. We show that even without observing payoffs, a player who knows the transition dynamics and observes only the opponents sequence of actions can still compete against an adversary who may have complete knowledge of the game. We formalize three distinct notions of symmetry in this setting and show that, under these conditions, the learning problem can be reduced to an instance of online learning, enabling the player to asymptotically match the return of the opponent despite lacking payoff observations. Our algorithms apply to both matrix and Markov games, and run in polynomial time with respect to the size of the game and the number of episodes. Our work broadens the class of games in which robust learning is possible under severe informational disadvantage and deepens the connection between online learning and adversarial game theory.

Equations

$$V^{{\pi }^1,{\pi }^2}(s):={\mathbb{E}}^{{\pi }^1,{\pi }^2}\left[\sum _{h^{\prime }=h}^{H}u(s_{h^{\prime }},a_{h^{\prime }}^1,a_{h^{\prime }}^2)|s_h=s\right],$$

$$\begin{gathered} V^{{\pi }^1,{\pi }^2}(s)={\mathbb{E}}^{{\pi }^1,{\pi }^2}[u(s,a_h^1,a_h^2) \\ +\sum _{s^{\prime }\in S_{h+1}}P(s^{\prime }\mid s,a_h^1,a_h^2)V^{{\pi }^1,{\pi }^2}(s^{\prime })|s_h=s]. \end{gathered}$$

$$D_t(a^1,a^2):=N_t(a^2,a^1)-N_t(a^1,a^2).$$

$$\alpha =\sum _{t=1}^T\sum _{h=1}^H{u}_{\star }(s_{h,t},a_{h,t}^1,a_{h,t}^2).$$

$$\left|\alpha \right|\le \sum _{h=1}^H\sum _{s\in S_h}\sum _{a^1,a^2\in A}\frac{1}{2}\left|D_{T+1}(s,a^1,a^2)\right|$$

$$(\mathbb{E}\left(\left|\alpha \right|\right){)}^2\le \mathbb{E}\left[{\alpha }^2\right]$$

$$\le \mathbb{E}\left[{\left(\sum _{h=1}^H\sum _{s\in S_h}\sum _{a^1,a^2\in A}\frac{1}{2}\left|D_{T+1}(s,a^1,a^2)\right|\right)}^2\right]$$

$$\le \mathbb{E}\left[\frac{|S|n^2}{4}\sum _{h=1}^H\sum _{s\in S_h}\sum _{a^1,a^2\in A}(D_{T+1}(s,a^1,a^2){)}^2\right]$$

$${\beta }_{t+1}^h-{\beta }_t^h$$

$$\le 2-4D_t(s_{h,t},a_{h,t}^1,a_{h,t}^2).$$

$$\sum _{h=1}^H{\beta }_{t+1}^h-{\beta }_t^h$$

$$=2H-4\sum _{h=1}^H{D}_t(s_{h,t},a_{h,t}^1,a_{h,t}^2).$$

$$(\mathbb{E}\left(\left|\alpha \right|\right){)}^2\le \mathbb{E}\left[\frac{|S|n^2}{4}\sum _{h=1}^H{\beta }_{T+1}^s\right]\le \frac{T|S|H{n}^2}{2}.$$

$$\mathbb{E}\left[\left|\sum _{t=1}^T\sum _{h=1}^H{u}_{\star }(s_{h,t},a_{h,t}^1,a_{h,t}^2)-u_{\star }(s_{h,t},a_{h,t}^2,a_{h,t}^1)\right|\right]$$

$$\le \sqrt{\frac{1}{2}T|S|H{n}^2}.$$

$$U=\{$$

$$\forall (s,a^1,a^2)\in S\times A\times A,\text{and}$$

$$V^{{\pi }^1,{\pi }^2}(s_1)=-V^{{\pi }^2,{\pi }^1}(s_1),\forall {\pi }^1,{\pi }^2\in {\Pi }_M\},$$

$${\pi }^1(s)=\{\begin{array}{ll}1,& s=s_1\\ 1,& s=s_2^A\\ 1,& s=s_2^B\end{array}$$

$$u(s_H,a^1,a^2)+u(s_H,a^2,a^1)=2u(s_H,1,1).$$

$$V^{{\pi }^1,{\pi }^2}(s)={\mathbb{E}}^{{\pi }^1,{\pi }^2}[u(s,a^1,a^2)$$

$$+\sum _{s^{\prime }\in S_{h+1}}\mathbb{P}(s^{\prime }\mid s,a^1,a^2)V^{{\pi }^1,{\pi }^2}(s^{\prime })]$$

$$={\mathbb{E}}^{{\pi }^1,{\pi }^2}[u(s,a^1,a^2)$$

$$+\sum _{s^{\prime }\in S_{h+1}}\mathbb{P}(s^{\prime }\mid s,a^1,a^2)\text{@mathmeasure}\text{big@size}1\text{big@size}(\text{vbox}to0.0ptu(s^{\prime },1,1)+{\hat{V}}^{{\pi }^1,{\pi }^2}(s^{\prime })\text{@mathmeasure}\text{big@size}1\text{big@size})\text{vbox}to0.0pt]$$

$$={\mathbb{E}}^{{\pi }^1,{\pi }^2}[\hat{u}(s,a^1,a^2)$$

$$+\sum _{s^{\prime }\in S_{h+1}}\mathbb{P}(s^{\prime }\mid s,a^1,a^2){\hat{V}}^{{\pi }^1,{\pi }^2}(s^{\prime })]$$

$$={\hat{V}}^{{\pi }^1,{\pi }^2}(s).\text{qed}$$

$${\pi }^1(z)=\{\begin{array}{ll}1,& z=(s_1)\\ 1,& z=(s_1,1,2,s_2)\\ 1,& z=(s_1,2,1,s_2)\end{array};$$

$${\pi }^2(z)=\{\begin{array}{ll}2,& z=(s_1)\\ 1,& z=(s_1,1,2,s_2)\\ 2,& z=(s_1,2,1,s_2)\end{array},$$

$$V^{{\pi }^1,{\pi }^2}(s_1)=u(s_1,1,2)+u(s_2,1,1)=1+0=1.$$

$$V^{{\pi }^2,{\pi }^1}(s_1)=u(s_1,2,1)+u(s_2,2,1)=-1-1=-2.$$

$$u^{\prime }(a^1,a^2)=u(s_1,a^1,a^2)$$

$$+\sum _{s^{\prime }\in S}P(s^{\prime }\mid s_1,a^1,a^2)V^{{\pi }^{\ast },{\pi }^{\ast }}((s_1,a^1,a^2,s^{\prime })),$$

$${\mathbb{E}}^{{\pi }^1,{\pi }^2}\left[u^{\prime }(a_1^1,a_1^2)\right]={\mathbb{E}}^{{\pi }^1,{\pi }^2}[u(s_1,a_1^1,a_1^2)$$

$$+\sum _{s^{\prime }\in S}P(s^{\prime }\mid s_1,a_1^1,a_1^2)\underset{=V^{{\pi }^1,{\pi }^2}((s_1,a_1^1,a_1^2,s^{\prime }))}{\underbrace{V^{{\pi }^{\ast },{\pi }^{\ast }}((s_1,a_1^1,a_1^2,s^{\prime }))}}]$$

$$=V^{{\pi }^1,{\pi }^2}(s_1).$$

$$u(a^1,a^2)=\{\begin{array}{ll}1& \text{if }{D}_T(a^2,a^1)\le 0\\ -1& \text{if }{D}_T(a^2,a^1)>0\end{array}$$

$$u(a^1,a^2)N(a^1,a^2)+u(a^2,a^1)N(a^2,a^1)=-\left|D_T(a^1,a^2)\right|$$

$$\mathbb{E}\left[\left|\sum _{t=1}^{T}u(a_t^1,a_t^2)\right|\right]$$

Full text

Machine Learning, ICML, Markov Games,Stochastic Games,Game Theory,Online Learning,Payoff-Free Learning,Optimization under uncertainty

1 Introduction

In unfamiliar or competitive settings, imitation often emerges as a natural coping tactic. In rapidly evolving online markets, where advertising tactics can shift daily, newer companies frequently align their campaigns with those of more seasoned competitors. Directly assessing the success of a marketing effort is challenging, so inexperienced firms may gain by replicating the strategies of those with greater market knowledge. Lacking the time or budget for extensive research, these newcomers must act decisively to stay competitive against entrenched players. However, imitation carries significant risk: executed well, it can lead to outsize gains with minimal investment; executed poorly, it can result in outcomes worse than random chance. This problem is even greater if one consider the scenario where company uses a long range advertising campaign where each advertising strongly dependent on previous tactics. To study this behavior, we frame it as a strategic interaction where the objectives of the competitor may conflict with your own. More specifically a zero-sum symmetric game.

Previously, Feldman et al. (2010) studied repeated play of finite zero-sum symmetric matrix games. They considered a scenario where the learner has no information about the game and does not observe any payoff. The learner is only aware of the available actions and observe the actions played by the adversary. On the other hand, the adversary may have complete prior knowledge of the payoff matrix. They demonstrated a simple copycat strategy that guarantees, against any opponent’s strategy: $\mathbb{E}[|\sum_{t=1}^{T}u(a^{1}_{t},a^{2}_{t})|]=O(n\sqrt{T})$, where $T$ is the number of rounds, $u$ is the payoff function, $a^{1}_{t},a^{2}_{t}$ are the actions played by the learner and adversary on round $t$, respectively, and $n$ is the number of actions.

In our work we ask the following natural question: can the copycat strategy of Feldman et al. (2010) be extended to Markov games, where the current state depends on the history of actions? In such a setting, how can a learner effectively mimic an adversary and ensure sublinear return?

Crucially, we operate under a severe informational disadvantage. We assume the learner must navigate the complex transition dynamics without possessing prior knowledge of the game’s parameters, neither the payoff function nor the underlying transition laws are known. The learner must rely solely on observing the actions played by the adversary.

To address this, our first challenge is to formalize the concept of a zero-sum symmetric Markov game. Intuitively, symmetry captures the property that if the players were to swap roles, exchanging their strategies and positions, their expected payoffs would simply be negated. This suggests a holistic definition where the value of the game remains symmetric over the entire horizon. Capturing this intuition at different levels of granularity, we identify three natural notions of symmetry: The first is per-state symmetric games (SSG), where each state corresponds to a symmetric zero-sum matrix game. The second, symmetry with respect to Markov policies (MSG), captures symmetry between any two Markov policies; namely policies that depend only on the current state. The third, symmetry with respect to history-dependent policies (HSG), generalizes this further to symmetry between any pair of history-dependent policies.

We aim to design a learning algorithm that works across all three settings. Crucially, we demonstrate that unlike symmetric matrix games, where a learner can simply mimic the opponent, Markov games introduce complex transition dynamics that the learner must navigate.

Our central theoretical insight is a counter-intuitive hierarchy of difficulty. One might expect the broader policy-based definitions (MSG and HSG) to generalize the local state-based definition (SSG). Instead, we prove that the constraints imposed by MSG and HSG are so strong that they reduce the problem’s complexity. Specifically, we show that any MSG can be transformed into an equivalent SSG, and any HSG reduces to a single symmetric matrix game. Consequently, we provide polynomial-time algorithms for these settings. To the best of our knowledge, our results are the first to demonstrate tractable learning in Markov games without observing payoffs, restricting the opponent, or requiring auxiliary information.

1.1 Related work

Symmetric Markov Games.

Several recent papers have studied how to formalize symmetry in Markov decision processes (MDPs) and Markov games. Zinkevich and Balch (2001) introduce a notion of symmetry in MDPs and show how it can be exploited for more effective learning in single-agent as well as multi-agent systems. In their formulation, two states are symmetric if they have symmetric actions, and two actions are symmetric if they lead to symmetric outcomes. They further extend this idea to symmetric MDPs, requiring both the payoff and transition functions to be symmetric. Zinkevich (2006) broaden this perspective to settings involving multiple agents.

Another line of work formulates symmetry at the level of payoffs in Markov games. Flesch et al. (2013) analyze a two-player Markov game in which the payoff functions are symmetric: if the players’ actions at a given state are exchanged, then their payoffs are swapped as well. A similar approach is taken by Yongacoglu et al. (2023), who consider a multi-player setting and require invariance under arbitrary permutations of the action sets. Both frameworks are flexible enough to capture zero-sum games. A key limitation, however, is that they assume transition dynamics are insensitive to the ordering of players’ actions, i.e., swapping the players does not alter the state dynamics, which restricts the generality of the resulting games.

Learning in Markov games. Markov games pose significant theoretical and algorithmic challenges, particularly in adversarial and partially observable settings. Prior work identified strong hardness results for core learning objectives in this domain. For instance, Bai et al. (2020) prove that computing a best response policy in zero-sum Markov games when facing an adversarial opponent is at least as difficult as the well-known parity with noise problem, which is widely believed to be computationally intractable.

A closely related special case that has been extensively studied is that of MDPs with known transitions but adversarial or unknown rewards. Such an MDP can be interpreted as a Markov game in which the adversary acts as a second player whose “actions” either modify the state payoffs or, equivalently, redirect transitions to payoff-altered versions of the next state. This line of research (e.g., Even-Dar et al., 2009; Zimin and Neu, 2013; Jin and Luo, 2020) primarily focuses on scenarios where the payoff function may vary over time. In a related line of work, Tian et al. (2021); Liu et al. (2022) establish tight lower bounds on regret in the online learning setting, showing that no algorithm can achieve better than $\Omega(\min\{\sqrt{2^{H}T},T\})$ regret in episodic zero-sum Markov games with unknown transitions, even when the adversary is limited to using Markov strategies.

Despite these limitations, recent advances have uncovered tractable settings under more cooperative assumptions. For example, Erez et al. (2023); Cai et al. (2024); Cui et al. (2023); Yang and Ma (2022); Jin et al. (2021) design decentralized algorithms for general-sum Markov games that achieve sublinear regret, assuming all players employ the same proposed learning rule. Their approach requires no communication between agents and guarantees convergence to a (coarse) correlated equilibrium.

2 Preliminaries

2.1 Markov games

A two-player zero-sum finite horizon Markov game is defined by the quintuple $(S,A,H,P,u)$, where $H\geq 1$ is the time horizon of the game. $S$ is a finite set of states partitioned as $S=\bigsqcup_{h=1}^{H}S_{h}$, where $S_{1}=\{s_{1}\}$. Without loss of generality, we assume that all states in $S$ are reachable from the initial state $s_{1}$ under some pair of policies, as unreachable states do not affect the game’s value or dynamics. $A=\{1,\dots,n\}$ is a finite set of actions available to both players; $P(\cdot\mid s_{h},a^{1},a^{2})\in\Delta(S_{h+1})$ is the probability distribution over the next state when the players 1 and 2 take actions $a^{1}$ and $a^{2}$ respectively at state $s_{h}\in S_{h}$. Finally, $u:S\times A\times A\rightarrow[-1,1]$ is the payoff function for player 1. As the game is zero-sum, $-u$ is the payoff function for player 2.

At every time step $h=1,\dots,H$, the players observe the current state $s_{h}$, then choose actions $a^{1}_{h},a^{2}_{h}$ respectively. The players choose their actions independently at random. Then player 1 gains $u(s_{h},a^{1}_{h},a^{2}_{h})$ and player 2 gains $-u(s_{h},a^{1}_{h},a^{2}_{h})$, and the game transitions to the next state $s_{h+1}$ drawn from $P(\cdot\mid s_{h},a^{1}_{h},a^{2}_{h})$. The game ends after time step $H$.

A history $z_{h}=(s_{1},a_{1}^{1},a_{1}^{2},\dots,s_{h-1},a_{h-1}^{1},a_{h-1}^{2},s_{h})$ is the sequence of states and actions generated by the players up to time $h$. We denote by $Z_{h}$ the set of histories up to time $h$. A (history-dependent) policy for player $i$ is a function $\pi^{i}:Z_{h}\mapsto\Delta(A)$ that maps each history $z_{h}$ to a probability distribution from which player $i$ draws their action when observing $z_{h}$. A policy is Markov if it is a function of only the current state, namely $\pi^{i}:S\mapsto\Delta(A)$.

We also introduce the concept of the value function, which captures the expected cumulative payoff from a given state onward and will be fundamental in defining optimality and proving performance guarantees. For policies $\pi^{1},\pi^{2}$ of the two players, time step $h$ and state $s\in S_{h}$, the value function is defined as:

[TABLE]

where the expectation is taken over the randomness in the actions taken by the policies and the transition dynamics. The value function satisfies the Bellman equations (see, e.g., Puterman, 1990):

[TABLE]

We define the value of the game as $V^{\star}=\max_{\pi^{1}}\min_{\pi^{2}}V^{\pi^{1},\pi^{2}}(s_{1})$. The value is guaranteed by a Markov safety level policy $\pi^{\star}$ for player 1, that is $V^{\pi^{\star},\pi^{2}}(s_{1})\geq V^{\star}$ for any strategy $\pi^{2}$ for player 2. Moreover, there is a polynomial-time dynamic programming algorithm that returns such $\pi^{\star}$ (Wal and Wessels, 1976).

2.2 The Copycat Strategy

To formalize the strategy, we first define the count matrix $N_{t}\in\mathbb{Z}^{n\times n}$, where $N_{t}(a^{1},a^{2})$ denotes the number of times the joint action $(a^{1},a^{2})$, the learner playing $a^{1}$ and the adversary playing $a^{2}$, has occurred up to time $t$ (noninclusive). We further define the discrepancy matrix $D_{t}$ as the difference between the counts of symmetric action pairs:

[TABLE]

The Copycat strategy (Feldman et al., 2010), restated below, is a simple mixed strategy that can be computed at period $t$ in time $\mathrm{poly}(n,\log t)$ using linear programming. The strategy’s goal is to equalize the number of occurrences of $(a^{1},a^{2})$ and $(a^{2},a^{1})$ for each $a^{1},a^{2}\in A$. To this end, the Copycat plays a virtual zero-sum game defined by $D_{t}$ in each period. Crucially, observe that by definition $D_{t}$ is skew-symmetric. This implies that when $D_{t}$ is treated as a payoff matrix, it defines a symmetric zero-sum game with a value of [math].

3 Problem definition and main results

We consider the problem of learning a two-player zero-sum Markov symmetric game by repeated play. We operate under a severe informational disadvantage: the learner never observes the game’s payoff, only the adversary’s actions, and assumes no prior knowledge of the transition probability function $P$. In contrast, we assume the adversary can be omniscient, possessing complete prior knowledge of both the transition dynamics $P$ and the payoff function $u$.

The game is played for $T$ episodes. At the beginning of each episode $t$, both players choose policies $\pi_{t}^{1},\pi_{t}^{2}$, that are then used to play the game in the current episode. While the opponent can choose any, possibly history-dependent, policy, we restrict the learner to select only Markov policies to facilitate efficient learning. Thereafter, both players observe the trajectory sampled from the game’s dynamics and the actions of their opponent. Crucially, only the adversary can calculate the incurred payoff. The goal of the learner is to guarantee their own payoff approaches the value of the game, which is 0.

Formally, the learner minimizes $\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left\lvert\vbox to0.0pt{}\right.}}}}{\mathbb{E}\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left[\vbox to0.0pt{}\right.}}}}{\sum_{t=1}^{T}V^{\pi_{t}^{1},\pi_{t}^{2}}(s_{1})}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left]\vbox to0.0pt{}\right.}}}}}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left\rvert\vbox to0.0pt{}\right.}}}}$. We aim for a bound that is sublinear in $T$, implying the average payoff converges to zero. In this work we consider three types of symmetry in Markov games.

Assumption 3.1** (Per-state symmetric game; SSG).**

A zero-sum Markov game is SSG if every state corresponds to a symmetric matrix game. Formally, $u(s,a^{1},a^{2})=-u(s,a^{2},a^{1})$ for all $s\in S$ and $a^{1},a^{2}\in A$.

While SSG enforces symmetry locally on the individual state level, one might ask: what if we only require the expected return to be symmetric?

Assumption 3.2** (Symmetry w.r.t. Markov policies; MSG).**

A zero-sum Markov game is MSG if for every Markov policies $\pi^{1},\pi^{2}$ it holds that $V^{\pi^{1},\pi^{2}}(s_{1})=-V^{\pi^{2},\pi^{1}}(s_{1}).$

Finally, we consider the most expansive definition, where symmetry must hold even for history-dependent policies.

Assumption 3.3** (Symmetry w.r.t. history-dependent policies; HSG).**

A zero-sum Markov game is HSG if for every two policies $\pi^{1},\pi^{2}$, possibly history-dependent, it holds that $V^{\pi^{1},\pi^{2}}(s_{1})=-V^{\pi^{2},\pi^{1}}(s_{1}).$

3.1 Our results

Our analysis reveals a counter-intuitive hierarchy across the three symmetry classes. One might naturally assume that “holistic” notions of symmetry (MSG and HSG), which enforce symmetry over policies or histories rather than local states, would yield a richer, more general class of games. However, we establish the exact opposite: stronger symmetry requirements impose such severe constraints on the game structure that they effectively simplify the learning landscape. We detail this hierarchy below:

First, for SSG games (3.1), we establish the following baseline guarantee.

For SSG games (3.1), we establish the following baseline guarantee.

Proposition 3.4**.**

In Section 4 we show that the learner can guarantees $\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left\lvert\vbox to0.0pt{}\right.}}}}{\mathbb{E}\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left[\vbox to0.0pt{}\right.}}}}{\sum_{t=1}^{T}V^{\pi_{t}^{1},\pi_{t}^{2}}(s_{1})}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left]\vbox to0.0pt{}\right.}}}}}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left\rvert\vbox to0.0pt{}\right.}}}}=O(n\sqrt{T|S|H})$ for SSG games in $\mathrm{poly}(|S|,n,T)$ time.

This result (detailed in Section 4) relies on the observation that local state symmetry allows the learner to decouple the Markov game. By treating each state as an isolated matrix game, we can deploy the Copycat strategy of (Feldman et al., 2010) independently at each step to achieve sublinear regret.

Next, we address MSG games (3.2). Here, the payoff structure is significantly more intricate, as the value depends on the coupled interaction between immediate rewards and future game dynamics. Intuitively, this dependency should make learning harder. However, we prove that the MSG symmetry condition is unexpectedly restrictive.

In Section 5, we unveil a reductive transformation that disentangles the game’s complex transition dynamics. We show that by identifying the correct transformation, we can recursively “peel back” the horizon, revealing that every layer of an MSG effectively decomposes into independent skew-symmetric matrix games. Crucially, we prove that this transformation preserves the game’s value for any pair of strategies. Consequently, from the learner’s perspective, the MSG class is structurally equivalent to SSG, enabling the direct and efficient application of our SSG algorithm.

For MSG games (3.2) we prove the following.

Proposition 3.5**.**

By Playing the SSG algorithm the learner can guarantees $\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left\lvert\vbox to0.0pt{}\right.}}}}{\mathbb{E}\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left[\vbox to0.0pt{}\right.}}}}{\sum_{t=1}^{T}V^{\pi_{t}^{1},\pi_{t}^{2}}(s_{1})}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left]\vbox to0.0pt{}\right.}}}}}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left\rvert\vbox to0.0pt{}\right.}}}}=O(n\sqrt{T|S|H})$ for MSG games.

Finally, for HSG games (3.3), we show that the symmetry constraint is so strong it trivializes the sequential nature of the problem. In Section 6, we prove that the HSG assumption forces the expected future value from time step $h=2$ onward to be constant, regardless of the players’ actions. This effectively collapses the Markov game into a single equivalent matrix game, which can be solved directly via the standard Copycat algorithm.

For HSG games (3.3) we prove the following.

Proposition 3.6**.**

In Section 6 we describe a strategy that guarantees $\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left\lvert\vbox to0.0pt{}\right.}}}}{\mathbb{E}\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left[\vbox to0.0pt{}\right.}}}}{\sum_{t=1}^{T}V^{\pi_{t}^{1},\pi_{t}^{2}}(s_{1})}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left]\vbox to0.0pt{}\right.}}}}}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left\rvert\vbox to0.0pt{}\right.}}}}=O(Hn\sqrt{T})$ for HSG games. Our algorithm runs in $\mathrm{poly}(n,T,|S|)$ time.

To complement these upper bounds, we provide a matching lower bound for the SSG setting in Appendix A.

Proposition 3.7**.**

There exists an adversary and a payoff function for which $\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left\lvert\vbox to0.0pt{}\right.}}}}{\mathbb{E}\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left[\vbox to0.0pt{}\right.}}}}{\sum_{t=1}^{T}V^{\pi_{t}^{1},\pi_{t}^{2}}(s_{1})}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left]\vbox to0.0pt{}\right.}}}}}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left\rvert\vbox to0.0pt{}\right.}}}}=\Omega(n\sqrt{T|S|H})$. This result confirms that our algorithms operate in the optimal regime with respect to the parameters of the game.

Summary. Our results generalize the Copycat scheme to the Markovian setting, establishing a comprehensive framework for learning in symmetric games without payoffs. Our analysis uncovers a fundamental trade-off: while MSG and HSG appear theoretically broader, their requirement for global value symmetry imposes such severe structural constraints that the learning problem effectively simplifies. By examining the most natural symmetry notions, we argue that for a symmetry notion to support a meaningful Copycat strategy, it must be defined at the state level (SSG). Our findings suggest that SSG is not merely the most intuitive definition, but structurally the most general; in contrast, attempting to induce complexity from the game’s dynamics through holistic payoff definitions effectively neutralizes the impact of the transition function, implicitly reducing the game to a simpler, often static, problem. Finally, our matching lower bound validates that our approach to the copycat problem is near-optimal. An interesting avenue for future work is the analysis of approximate symmetry. While our work establishes the theoretical foundations for SSG, MSG, and HSG settings, investigating how these guarantees hold under small deviations from perfect symmetry would further bridge the gap to practical applications.

4 Copycat strategy in SSG

In this section, we prove Proposition 3.4 and present our strategy for learning in Per-state Symmetric Games (SSG) under 3.1. In this setting, every state $s\in S$ encapsulates a symmetric zero-sum matrix game.

4.1 Strategy: Copycat on Each State

At first glance, generalizing from matrix games to Markov games suggests that the adversary might exploit the transition dynamics to gain an advantage. Since players’ actions influence future state visitations, one might assume a holistic approach accounting for long-term planning is required. However, we demonstrate that the Copycat strategy is surprisingly robust to unknown dynamics. We show that for SSG, the learner can effectively decouple the transition dynamics from the immediate payoffs. By simply treating each state as an independent matrix game and applying the Copycat strategy locally, the learner can ignore the transition function $P$ entirely while still guaranteeing sublinear regret, avoiding any exponential scaling with the game’s parameters.

We analyze the strategy where the learner plays the original Copycat algorithm (Feldman et al., 2010) on each state $s\in S$ independently. Let $u_{\star}$ denote the true (unobserved) payoff of the game. We define the cumulative payoff $\alpha$ as:

[TABLE]

Using the fact that the magnitude of the cumulative payoff is bounded by the final discrepancy of the action counts. Specifically:

[TABLE]

where $D_{T+1}(s,a^{1},a^{2})$ is the final entry in the discrepancy matrix for state $s$ after $T$ episodes.

and,by Cauchy-Schwartz:

[TABLE]

To analyze the growth of this term, we define the potential function $\beta^{h}_{t}$ representing the sum of squared discrepancies at time $t$ for states at horizon $h$: $\beta^{h}_{t}=\sum_{s\in S_{h}}\sum_{a^{1},a^{2}\in A}(D_{T+1}(s,a^{1},a^{2}))^{2}.$

We analyze the single-step difference $\beta^{h}_{t+1}-\beta^{h}_{t}$. Note that in each episode $t$, exactly one state $s_{h}$ is visited at each step $h$. For the unvisited states, the discrepancy does not change. For the visited state $s=s_{h}$, the change is bounded as follows:

[TABLE]

Summing over the horizon $H$:

[TABLE]

Crucially, the Copycat strategy computes the max-min strategy for the virtual game defined by $D_{t}(s,\cdot,\cdot)$. This guarantees that the expected “virtual payoff” is non-negative at every step: $\mathbb{E}\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left[\vbox to0.0pt{}\right.}}}}{D_{t}(s_{h},a^{1}_{t},a^{2}_{t})}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left]\vbox to0.0pt{}\right.}}}}\geq 0.$ By linearity of expectation: $\mathbb{E}\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left[\vbox to0.0pt{}\right.}}}}{\sum_{h=1}^{H}D_{t}(s_{h,t},a^{1}_{h,t},a^{2}_{h,t})}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left]\vbox to0.0pt{}\right.}}}}=\sum_{h=1}^{H}\mathbb{E}\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left[\vbox to0.0pt{}\right.}}}}{D_{t}(s_{h,t},a^{1}_{h,t},a^{2}_{h,t})}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left]\vbox to0.0pt{}\right.}}}}\geq 0.$

Substituting this back into the potential function inequality, we obtain: $\mathbb{E}\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left[\vbox to0.0pt{}\right.}}}}{\sum_{h=1}^{H}\beta^{h}_{t+1}-\beta^{h}_{t}}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left]\vbox to0.0pt{}\right.}}}}\leq 2H.$

Since $\beta^{h}_{0}=0$ (assuming zero initial discrepancy), telescoping the sum over $T$ yields $\mathbb{E}\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left[\vbox to0.0pt{}\right.}}}}{\sum_{h=1}^{H}\beta^{h}_{T+1}}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left]\vbox to0.0pt{}\right.}}}}\leq 2HT$. Finally, substituting this bound into our expression for the squared expected payoff:

[TABLE]

Taking the square root yields the desired bound. This result confirms that in the SSG setting, the adversary’s ability to influence the trajectory yields minimal advantage, as they cannot force a worse regret than what is incurred by solving the local matrix games.

4.1.1 General sum case

We extend our results to general-sum symmetric games, a significantly more challenging setting where the value of the game is not necessarily zero (i.e., outcomes exists where both players win or both lose).

We achieve this by considering the normalized “skew-symmetrized” payoff $\hat{u}(s,a^{1},a^{2})=\frac{1}{2}(u_{\star}(s,a^{1},a^{2})-u_{\star}(s,a^{2},a^{1}))$ , such that $\hat{u}(s,a^{1},a^{2})\in[-1,1]$. The learner’s payoff with respect to this contracted game is similarly bounded:

[TABLE]

This implies a powerful result: that even in general-sum SSG, the learner can enforce a symmetric outcome on average regardless of the opponent’s strategy. Namely, even if the adversary plays sub-optimally or maliciously, the learner effectively “mirrors” the opponent’s success (or failure), ensuring they are never exploited.

In the next section, we generalize our approach to the broader class of MSG games (3.2).

5 Copycat strategy in MSG

In this section, we prove Proposition 3.5. Our approach to learning under the MSG assumption (3.2) focuses on exploiting the structural constraints induced by the symmetry of the value function.

The key distinction from SSG is that symmetry is now enforced on the expected cumulative payoff rather than locally at each state. The set of valid MSG payoff functions is defined as:

[TABLE]

here $\Pi_{M}$ denotes the set of all Markov policies.

The central challenge in the MSG setting is that while the game is globally zero-sum symmetric, individual states need not be skew-symmetric. This breaks the local guarantee of the Minimax strategy used in the SSG case: the expected “virtual payoff” at any given step is no longer guaranteed to be non-negative.

Intuitively, one might expect the MSG assumption to be a broad generalization of SSG. However, we demonstrate that this intuition is misleading: the holistic nature of MSG imposes constraints that some locally symmetric games fail to satisfy. We illustrate this distinction with the counter-example in Figure 1.

In this example, if $P(s_{2}^{A}\mid s_{1},1,2)=P(s_{2}^{B}\mid s_{1},2,1)=1$, consider the deterministic strategies:

[TABLE]

Where $\pi^{i}:S\mapsto A$ denotes the action selected by player $i$ at each state. Here, $V^{\pi^{1},\pi^{2}}(s_{1})=1+0=1$, whereas $V^{\pi^{2},\pi^{1}}(s_{1})=-1+1=0$. The game value is asymmetric, violating MSG, despite every local payoff matrix being skew-symmetric.

This counter-example clearly demonstrates that the MSG assumption induces severe structural constraints. When players swap policies (switching $\pi^{1}$ and $\pi^{2}$), the resulting trajectory of state visitations may change completely. Yet, the MSG condition demands that the total value of the game remains exactly symmetric (negated) despite this potentially drastically different path through the state space. This implies that the payoffs and transition dynamics must be tightly coupled to ensure such invariance holds across all possible Markov policies.

We leverage this insight by analyzing the game from the latest timestep, by fixing the strategy up to the final state. This motivates our recursive reduction technique.

5.1 Expected Payoff Redistribution

We introduce the Expected Payoff Redistribution (EPR) transformation, which propagates expected future rewards into the current state to reveal the underlying symmetry of the future states.

Definition 5.1** (Expected Payoff Redistribution; EPR).**

Given a payoff function $u$ with horizon $H\geq 2$, the EPR transformation on round $h$ yields a new payoff $\hat{u}$ defined as:

• •

$\hat{u}(s,a^{1},a^{2})=u(s,a^{1},a^{2})+\sum_{s^{\prime}\in S_{h+1}}\mathbb{P}(s^{\prime}\mid s,a^{1},a^{2})u(s^{\prime},1,1)$ for all $s\in S_{h}$.

• •

$\hat{u}(s,\cdot,\cdot)=u(s,\cdot,\cdot)-u(s,1,1)$ for all $s\in S_{h+1}$.

• •

$\hat{u}(s,\cdot,\cdot)=u(s,\cdot,\cdot)$ otherwise.

Lemma 5.2** (EPR on MSG).**

Consider an MSG game with $H\geq 2$. Applying the EPR transformation (Definition 5.1) on round $H-1$ yields $\hat{u}$ with the following properties:

At round $H$, the payoff matrix for every state is skew-symmetric. 2. 2.

If we truncate the last round, the resulting game of length $H-1$ satisfies the MSG assumption (3.2).

The intuition behind this result relies on “calibrating” the game using a reference action pair. The MSG assumption (3.2) mandates that $V^{\pi^{1},\pi^{2}}(s_{1})=-V^{\pi^{2},\pi^{1}}(s_{1})$ for any pair of Markov policies.

To isolate the payoff structure at the final step $H$, we employ a calibration technique. We analyze the game’s value under a reference strategy where, upon reaching a state $s\in S_{H}$, both players play a fixed action (e.g., action 1). Since the MSG assumption applies to all policies, the value generated by this specific behavior serves as a fixed baseline value.

By measuring the value of arbitrary actions relative to this baseline, we effectively subtract out the influence of the transition history. This reveals a rigid structure in the final layer where, for all $a^{1},a^{2}\in A$:

[TABLE]

We find that while the payoff matrix at $H$ is not necessarily skew-symmetric on its own, it becomes skew-symmetric once we subtract the baseline value of playing $(1,1)$.

The EPR transformation explicitly subtracts this $(1,1)$ baseline from the last layer and redistributes it to the previous layer. Once this “bias” is removed, the remaining payoff matrix at step $H$ is forced to be strictly skew-symmetric. Crucially, this redistribution allows us to propagate the baseline value to the previous round without breaking the MSG assumption for the shortened game. The complete proof is quite involved, and is deferred to Appendix D.

Lemma 5.3**.**

For any payoff function $u$ with $H\geq 2$, applying EPR (Definition 5.1) at any round $h<H$ preserves the value of the game. Specifically, for any strategies $\pi^{1},\pi^{2}\in\Pi_{M}$: $V^{\pi^{1},\pi^{2}}(s_{1})=\hat{V}^{\pi^{1},\pi^{2}}(s_{1}),$ where $\hat{V}$ is the value function under $\hat{u}$.

Proof.

Since $\hat{u}$ differs from $u$ only at steps $h$ and $h+1$, it suffices to show that value conservation holds locally. For any $s\in S_{h}$:

[TABLE]

5.2 MSG is Equivalent to SSG

Lemma 5.4**.**

For any MSG game with payoff $u$, there exists an SSG game with payoff $\hat{u}$ such that for all Markov policies: $V^{\pi^{1},\pi^{2}}(s_{1})=\hat{V}^{\pi^{1},\pi^{2}}(s_{1}).$

Proof.

We prove this by recursive reduction. We start by applying the EPR transformation (Definition 5.1). Lemma 5.2 guarantees that the transformed game, when viewed as a game of horizon $H-1$, retains the MSG property. Simultaneously, it ensures the payoff at the discarded $H$-th time step becomes strictly skew-symmetric.

We repeat this process $H-1$ times. At each step, we decouple the final time step, transforming its local payoff matrices into skew-symmetric form, and propagate the residual asymmetry to the preceding layer. Lemma 5.3 ensures that the value function remains invariant throughout. In the final step, the remaining payoff matrix at $s_{1}$ must itself be skew-symmetric to satisfy the MSG assumption (3.2). The result is a fully skew-symmetric game (SSG) that is equivalent to the original MSG. ∎

Consequently, from the learner’s perspective, the MSG class is structurally equivalent to SSG. The learner can apply the SSG algorithm directly to any MSG instance without modification. Next, we explore learning under the even stricter condition of 3.3.

6 Copycat strategy in HSG

In this section we prove Proposition 3.6. We show that under the strict HSG assumption (3.3), the Markov game effectively collapses into a single symmetric zero-sum matrix game. This reduced game can be solved using the Copycat strategy (Algorithm 1), achieving a cumulative payoff bound of $O(Hn\sqrt{T})$ in polynomial time.

Just as not every SSG is an MSG, it is crucial to recognize that the HSG class is a strictly smaller subset of MSG. Intuitively, HSG seems like a natural generalization, but the requirement for symmetry over history-dependent policies rules out games with complex state transition structure.

We illustrate this distinction with the counter-example in Figure 2: a game that satisfies the MSG condition but fails to be an HSG.

Consider the game above with deterministic transitions $P(s_{2}\mid s_{1},\cdot,\cdot)=1$. Let us examine the value under specific history-dependent strategies:

[TABLE]

where $\pi^{i}:Z\mapsto A$ denotes the action selected by player $i$ for each history.

Under these strategies, the trajectory is $(s_{1},1,2)\to s_{2}$. The value is:

[TABLE]

However, if we swap roles, the trajectory becomes $(s_{1},2,1)\to s_{2}$, and the opponent responds differently to this history. The value becomes:

[TABLE]

Since $1\neq-(-2)$, the game violates HSG. This example demonstrates that HSG requires symmetry to hold even when strategies condition their future behavior on the specific history of past actions. This restriction is so severe that it effectively forces the future value to be determined solely by the history.

To formalize this, we frame the HSG as an MSG defined over the space of histories (see Appendix E). Using this rearrangement, we establish a powerful structural property:

Lemma 6.1** (All Value Functions Are Equal).**

Let $z_{2}\in Z$ be a history up to time 2, namely of the form $z_{2}=(s_{1},a^{1}_{1},a_{1}^{2},s_{2})$. Let $\Pi_{H}$ denotes the set of all history-dependent policies. For any $\pi^{1},\pi^{2},{\pi^{1}}^{\prime},{\pi^{2}}^{\prime}\in\Pi_{H}$, the value at $z_{2}$ is identical: $V^{\pi^{1},\pi^{2}}(z_{2})=V^{{\pi^{1}}^{\prime},{\pi^{2}}^{\prime}}(z_{2}).$

The complete proof is provided in Appendix F. This lemma implies that from time step 2 onward, the expected return is a fixed constant, completely independent of the players’ future strategies. Consequently, the entire outcome of the game is determined solely by the actions played in the first round.

The intuition behind this collapse is that history-dependent policies are extremely powerful, they can condition their future behavior on the exact outcome of the first round. If the value function from step 2 onward depended on the specific policies chosen, we could construct a pair of “sensitive” strategies that exploit this dependency to violate the global symmetry condition. The rigid requirement that symmetry must hold for any pair of history-dependent policies removes all degrees of freedom from the future game dynamics. The only way to satisfy this constraint is if the future value is effectively pre-determined, rendering the players’ subsequent choices irrelevant to the expected payoff. This observation allows us to reduce the entire HSG to a single matrix game.

Lemma 6.2** (Equivalent Matrix Game).**

For any HSG with payoff function $u$, there exists an equivalent matrix game with payoff function $u^{\prime}$ such that:

[TABLE]

where $\pi^{*}\in\Pi_{H}$ maintains $\mathbb{P}^{\pi^{*}}[a=1\mid z]=1$, $\forall z\in Z$.

Proof.

Given any pair of history-dependent policies $\pi^{1},\pi^{2}\in\Pi_{H}$, we compute the expected utility under $u^{\prime}$:

[TABLE]

Hence, the matrix game faithfully reproduces the expected value of the original HSG. ∎

Since the problem reduces to a repeated matrix game, the learner can simply apply the standard Copycat strategy (Algorithm 1) to $u^{\prime}$, guaranteeing the stated regret bound.

7 Acknowledgments

This project is supported by the Israel Science Foundation (ISF, grant number 2250/22).

Impact Statement

This paper presents work whose goal is to advance the field of Machine Learning, specifically in the domain of Multi-Agent Reinforcement Learning and Game Theory. By developing algorithms for imitation in competitive Markovian environments, our work addresses the challenge of decision-making under severe informational disadvantage. Theoretically, this lowers the barrier to entry in complex competitive markets (e.g., automated advertising), enabling resource-constrained agents to perform robustly against sophisticated incumbents without requiring extensive knowledge of the system dynamics. We do not foresee any immediate negative ethical or societal consequences that must be specifically highlighted here.

Appendix A Lower Bound Analysis

Since the learner does not observe payoffs, they are compelled to play a strategy that is robust against any potential payoff function. This setting is analytically equivalent to a scenario where the adversary selects the worst-case payoff function after the game concludes. We begin by analyzing the fundamental case of a repeated matrix game.

Consider the following adversarial payoff function, defined in hindsight based on the final action discrepancies:

[TABLE]

Observe that for this specific payoff function, the cumulative contribution of any symmetric pair $(i,j)$ satisfies:

[TABLE]

Substituting this into the expected payoff guarantee yields:

[TABLE]

Thus, to lower bound the payoff, it suffices to lower bound the expected total discrepancy $\mathbb{E}\mathopen{}\mathclose{{\left[{\sum_{a^{1}<a^{2}}\mathopen{}\mathclose{{\left\lvert{D_{T}(a^{1},a^{2})}\right\rvert}}}\right]}}$

A.1 Matrix game

We establish a lower bound of $\Omega(n\sqrt{T})$ on the expected cumulative regret. We assume that the number of actions is fixed and satisfies $n>2$, that the time horizon $T$ is sufficiently large, and that the adversary plays a uniform strategy, choosing each action with probability $1/n$ independently at every step.

Let $N_{t}(a)$ denote the number of times the learner plays action $a$ up to time $t$. We analyze two regimes based on the learner’s action frequencies.

Case 1: The Unbalanced Regime

Suppose there exists an action $a^{1}$ such that the learner’s expected count deviates significantly from uniformity:

[TABLE]

To lower bound the regret, we decompose the discrepancy using the Reverse Triangle Inequality:

[TABLE]

First, we bound the Total Signal. Using the inequality $\sum_{a^{2}\in A}|N_{T}(a^{1})-N_{T}(a^{2})|\geq n|N_{T}(a^{1})-\frac{T}{n}|$, we obtain:

[TABLE]

Next, we bound the Total Noise. We define the noise term $\mathcal{E}_{t}$ as:

[TABLE]

The total noise corresponds to the magnitude of the sum of these variables. By applying Jensen’s inequality and bounding the variance, we have:

[TABLE]

Summing this bound over all $a^{2}\neq a^{1}$ and applying the Cauchy–Schwarz inequality to the sum of $n-1$ terms yields:

[TABLE]

Conclusion. Combining the signal bound with the noise bound above yields

[TABLE]

which holds for all sufficiently large $T$.

Case 2: The Balanced Regime

Suppose that for all actions $a\in A$, the counts are concentrated near the uniform allocation:

[TABLE]

We note that by Jensen’s Inequality:

[TABLE]

We lower-bound using the specific payoff of a Rock-Paper-Scissors (RPS) game on the cycle $\Delta=\{a^{1},a^{2},a^{3}\}$. Define the payoff function $u(a^{1}_{t},a^{2}_{t})$ as $1$ if the pair $(a^{1}_{t},a^{2}_{t})$ is a winning edge for the learner in $\Delta$, $-1$ if it is a losing edge, and [math] otherwise.

Let $X\coloneqq\sum_{t=1}^{T}u(a^{1}_{t},a^{2}_{t})$. By the triangle inequality, the total discrepancy on the cycle upper-bounds the magnitude of this cumulative payoff:

[TABLE]

To lower-bound $\mathbb{E}[\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left\lvert\vbox to0.0pt{}\right.}}}}{X}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left\rvert\vbox to0.0pt{}\right.}}}}]$, we employ the Fourth Moment Method (a consequence of the Hölder’s inequality):

[TABLE]

The Second Moment. The sequence $(u(a^{1}_{t},a^{2}_{t}))_{t=1}^{T}$ forms a martingale difference sequence. The conditional variance at step $t$ is non-zero only if the learner plays an action $i_{t}\in\Delta$ and the adversary (playing uniformly) hits a neighbor in the cycle (with probability $2/n$). Summing over all time steps:

[TABLE]

where $N_{\Delta}=\sum_{a\in\Delta}N_{T}(a)$ denotes the total number of times the learner plays in the cycle.

The Fourth Moment. By the Burkholder–Davis–Gundy inequality (with $p=4$):

[TABLE]

(Note: The last inequality follows because $\mathbb{E}[u^{4}]=\mathbb{E}[u^{2}]$ for indicator-like variables, and the cross-terms decouple into the product of their conditional variances: $\frac{2}{n}\times\frac{2}{n}=\frac{4}{n^{2}}$).

Since $N_{\Delta}$ scales linearly with $T$ (specifically $\mathbb{E}[N_{\Delta}]\propto T/n$), the quadratic term $\mathbb{E}[N_{\Delta}^{2}]$ grows as $T^{2}$, dominating the linear term $\mathbb{E}[N_{\Delta}]$. Thus, for sufficiently large $T$, we can absorb the linear term into the quadratic term:

[TABLE]

Bounding the Second Moment. We show that the second moment $\mathbb{E}[N_{\Delta}^{2}]$ is dominated by the squared mean. Since $N_{\Delta}$ and $\mathbb{E}\mathopen{}\mathclose{{\left[{N_{\Delta}}\right]}}$ are both restricted to the interval $[0,T]$, their distance is at most $T$. Thus, almost surely:

[TABLE]

Taking expectations and applying the regime condition $\mathbb{E}\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left[\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left[\vbox to0.0pt{}\right.}}}}{\mathopen{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left\lvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left\lvert\vbox to0.0pt{}\right.}}}}{N(a)-T/n}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left\rvert\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left\rvert\vbox to0.0pt{}\right.}}}}<n\sqrt{T}}\mathclose{\mathchoice{{\@mathmeasure{}{\big@size 1\big@size\displaystyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 1\big@size\textstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.7\big@size\scriptstyle\left]\vbox to0.0pt{}\right.}}}{{\@mathmeasure{}{\big@size 0.5\big@size\scriptscriptstyle\left]\vbox to0.0pt{}\right.}}}}$:

[TABLE]

The variance scales as $O(T^{1.5})$, which is asymptotically negligible compared to $\mathbb{E}\mathopen{}\mathclose{{\left[{N_{\Delta}}\right]}}^{2}\propto T^{2}$. Implying that for large enough $T>n^{6}$ we will have

[TABLE]

Completing the Bound. We now substitute these moments into the Hölder’s ratio. Squaring both sides implies:

[TABLE]

Substituting the second moment bound into the denominator:

[TABLE]

Finally, taking the square root imply:

[TABLE]

Aggregation. Summing over all $\binom{n}{3}$ cycles and dividing by the overcounting factor $(n-2)$ (since each edge is part of $n-2$ triangles):

[TABLE]

A.2 SSG games

We extend the lower bound to the Markovian setting by constructing a specific SSG instance with $|S|$ states and episodes of length $H$. Consider a game where the transition dynamics are uniform and independent of the current state or actions. Specifically, for any state $s$, step $h$, and action pair $(a^{1},a^{2})$:

[TABLE]

In this construction, the expected visitation frequency of any state is a random variable determined solely by the transition, independent of the players’ strategies. Let $T_{s}$ denote the random number of times state $s$ is visited over the entire game horizon of $T$ episodes. The expected number of visits is $\mathbb{E}[T_{s}]=\frac{HT}{|S|}$.

Since the states are decoupled by the uniform transitions, the game effectively decomposes into $|S|$ independent repeated matrix games, where the effective time horizon for state $s$ is $T_{s}$. Applying the reduction from Equation 1 locally to each state:

[TABLE]

We assume that $T$ is sufficiently large (specifically $T>\Omega(n^{6}|S|)$) such that for every state $s$, the effective horizon $T_{s}$ falls into the regime where the matrix game lower bound holds. Summing the lower bounds for each state:

[TABLE]

Appendix B Matrix game as Blackwell’s Approachability

Blackwell Approachability is an extension of Von Neumann’s minimax theorem (Von Neumann, 1928) to vector-valued payoffs. The objective is to ensure, in repeated play, that the average payoff vector converges to a desirable convex target set $K$. Unlike in the scalar case, fixed (oblivious) strategies may fail to guarantee convergence. Blackwell’s celebrated Approachability Theorem (Blackwell, 1956) shows that in the repeated game setting, an adaptive strategy that reacts to the opponent’s play can still guarantee convergence to $K$.

We first define the main concepts of approachability theory. A Blackwell instance is a tuple $(\mathcal{X},\mathcal{Y},L,U)$ such that $\mathcal{X}\subset\mathbb{R}^{n}$ is the action set of the learner, $\mathcal{Y}\subset\mathbb{R}^{m}$ is the action set of the adversary. Both action sets are convex and compact. The set $U\subset\mathbb{R}^{d}$ is convex and closed, and the vector valued function $L:\mathcal{X}\times\mathcal{Y}\mapsto\mathbb{R}^{d}$ is bilinear.

Definition B.1** (Approachable set).**

Given a Blackwell instance $(\mathcal{X},\mathcal{Y},L,U)$, we say that $U$ is approachable if there exists an algorithm ALG that selects points in $x^{1},\dots,x^{T}\in\mathcal{X}$ such that, for any sequence $y^{1},y^{2},\dots y^{T}\in\mathcal{Y}$, we have:

[TABLE]

where $x^{t}\leftarrow\textsc{ALG}(y^{1},y^{2},\dots y^{t-1})$.

Finally, we state Blackwell’s Approachability Theorem.

Theorem B.2**.**

*(Approachability Theorem; Blackwell, 1956)

For any Blackwell instance $(\mathcal{X},\mathcal{Y},L,U)$, $U$ is approachable if and only if $\;\forall y\in\mathcal{Y}\;\;\exists x\in\mathcal{X}\;:L(x,y)\in U$.*

B.1 Formulation of the copycat problem as Blackwell’s Approachability

We next formulate the copycat problem as a Blackwell’s Approachability problem. We define a Blackwell instance $(A,A,L,U)$, such that $A=\{1,\dots,n\}$ is the action set of both players. The target set $U$ is the space of all symmetric matrices:

[TABLE]

We define $L(a^{1},a^{2})$ as a matrix in $\mathbb{R}^{n\times n}$ whose entries are 0 except for a single entry $a^{1},a^{2}$ that equals 1.

Lemma B.3**.**

The copycat problem is approachable.

Proof.

For each $a^{2}$ we choose $a^{1}=a^{2}$ we get $L(i,j)=0$ for any $(i,j)$, except $L(a^{2},a^{2})=1$, which mean that $L\in U$. ∎

Lastly, consider a repeated symmetric matrix game with payoff $u_{\star}$. Then, since the problem is approachable:

[TABLE]

Appendix C Copycat type games are an Instance of Blackwell Approachability

Prior work by (Abernethy et al., 2011; Shimkin, 2016) show that Blackwell Approachability and Online Convex Optimization (see Appendix G), are algorithmically equivalent in the context of repeated games. Specifically, any algorithm that achieves approachability can be transformed into an online algorithm with low regret, and vice versa.

In the copycat setting, the learner does not observe the realized payoffs but knows that the payoff function belongs to a specific set $U$. The objective is to guarantee performance comparable to the optimal strategy for any payoff function within this set. We leverage the equivalence mentioned above to formalize this problem as an instance of Blackwell Approachability. This formulation allows us to derive a general solution that is independent of the specific structure of $U$.

While we detail the full Blackwell formulation for the matrix game case in Appendix B, for the general case we proceed by directly reducing the problem to online linear optimization.

We now present the main algorithm:

We conceptualize the algorithm as involving a third ”virtual player” who observes the interaction between the learner and the adversary. At the beginning of each round $t$, this virtual player selects a payoff function $u_{t}\in U$.

The virtual player’s goal is to minimize their cumulative loss. Given a realized trajectory $\tau=(s_{1},a_{1}^{1},a_{1}^{2},\dots,s_{H},a_{H}^{1},a_{H}^{2})$, the loss for episode $t$ is defined as the total payoff under the selected function: $\sum_{h=1}^{H}u_{t}(s_{h},a_{h}^{1},a_{h}^{2})$

To optimize this objective, the virtual player employs an online linear optimization algorithm ALG over the set $U$. At the end of round $t$, ALG receives a feedback loss tensor $l_{t}\in\mathbb{R}^{|S|\times n\times n}$. This tensor acts as a sparse indicator of the trajectory: its entries are set to $1$ for the $H$ state-action triplets visited in $\tau$ and [math] otherwise (corresponding to Line 6 in Algorithm 2).

In this manner we represent the loss as a linear function of the payoff matrix: $l_{t}\cdot u_{t}=\sum_{h=1}^{H}u_{t}(s_{h},a_{h}^{1},a_{h}^{2})$.

Note that $\|u\|_{F}\leq n\sqrt{|S|}$ for all $u\in U$ and that $\|l_{t}\|_{F}\leq\sqrt{H}$ for all $t=1,\dots,T$. Using OGD as our online algorithm ALG yields:

[TABLE]

The learner’s strategy is to select a safety-level strategy $\pi_{t}^{1}$ based as if $u_{t}$ is the payoff function for the game. As $u_{t}\in U$, the value of the game is zero, guaranteeing a non-negative expected payoff for the learner:

[TABLE]

regardless of the strategy $\pi_{t}^{2}$ chosen by the adversary. Now indeed suppose that the virtual player uses OGD. Let $u_{\star}$ represent the actual payoffs of the game. We bound the learner’s playoff as follows:

[TABLE]

Furthermore, we claim that Algorithm 2 runs in polynomial time. Indeed, computing a safety-level strategy is known to be polynomial (Wal and Wessels, 1976), and the remaining steps of the algorithm are similarly efficient. Thus, the runtime analysis reduces to ensuring that OGD over $U$ runs in polynomial time, which suffices to show that the Euclidean projection onto $U$ is efficient. For SSG games (3.1), this projection is trivial.

Appendix D Proof of Lemma 5.2

Proof.

We look at $\pi^{1},\hat{\pi}^{1},\pi^{2},\hat{\pi}^{2}\in\Pi_{M}$ under the special case where $\pi^{1}=\hat{\pi}^{1}$ and $\pi^{2}=\hat{\pi}^{2}$ for all $h$ except $H$ and for all states except $s\in S_{H}$. In this case:

[TABLE]

3.2 implies:

[TABLE]

We combine both observations to get:

[TABLE]

Now we set $\pi^{1},\pi^{2}$ such that they will always choose actions 1 in $s\in S_{H}$. Formally, $\mathbb{P}^{\pi^{1}}[a_{H}=1\mid s_{H}=s]=\mathbb{P}^{\pi^{2}}[a_{H}=1\mid s_{H}=s]=1$. Plugging these policies into Equation 5 gets:

[TABLE]

Next we choose $\pi^{1},\pi^{2}$ that choose actions uniformly at random everywhere except in $s\in S_{H}$. Formally: $\mathbb{P}^{\pi^{i}}[a=a_{h}\mid s_{h}]=\frac{1}{|A|}$ for all $h\in[H]$, $s_{h}\in S_{h}\setminus\{s\}$ and $i=1,2$. By the choice of our policies, we have that $\mathbb{P}^{\pi^{1},\pi^{2}}[s_{H}=s]=\mathbb{P}^{\pi^{2},\pi^{1}}[s_{H}=s]>0$, from which

[TABLE]

then for any discrete $\hat{\pi}^{2},\hat{\pi}^{1}$ this complete the proof for Item 1.

Now we rearrange Equation 6:

[TABLE]

For simplicity we will define:

[TABLE]

We begin our proof by considering the payoff at step $H$ for policies $\pi_{1},\pi_{2}\in\Pi_{M}$:

[TABLE]

Expanding the last two summands using the Markov property:

[TABLE]

Finally, substituting this back into the original expression for the payoff:

[TABLE]

We can clearly see that the left side must hold 3.2 hence the right side must as well, completing our proof.

∎

Appendix E Rearranging The HSG Assumption

In order to simplify our derivation, we formalize the HSG game differently. We create a new state space $Z$ in which every state represent a history in the original HSG, namely $Z=\bigsqcup_{h=1}^{H}Z_{h}$, where $Z_{h}=\{(s_{1},a_{1}^{1},a_{1}^{2}\dots s_{h}):\forall s_{1}\dots s_{h}\in S\;\text{and}\;a_{1}^{1},a_{1}^{2}\dots a_{h-1}^{1},a_{h-1}^{2}\in A\}$. The action space and the horizon remains the same. The initial state is $z_{1}=(s_{1})$. The dynamics in the new game are111We overload the notation $P$ and $u$ using histories to refer to the those of the new game.

[TABLE]

Lastly, the modified payoff function is:

[TABLE]

The definition of the value function of the new game follows immediately from our reduction. Lastly, since history-dependent policies in the original game are Markov in the new game, we can analyze our new rearranged HSG game similarly to the analysis done on MSG.

Appendix F Proof of Lemma 6.1

Lemma* (Restatement of Lemma 6.1).*

Let $z_{2}\in Z_{2}$ be a history up to time 2, namely of the form $z_{2}=(s_{1},a^{1}_{1},a_{1}^{2},s_{2})$. Let $\Pi_{H}$ denotes the set of all history-dependent policies. For any $\pi^{1},\pi^{2},{\pi^{1}}^{\prime},{\pi^{2}}^{\prime}\in\Pi_{H}$, the value at $z_{2}$ is identical:

[TABLE]

Before we start the proof of Lemma 6.1, for clarity, define the following shorthand for the expected utility at stage 1 under a pair of strategies:

[TABLE]

Proof.

We start by analyzing the HSG structure under strategies $\pi^{1},\pi^{2}$ at the second stage:

[TABLE]

Lets start our analysis with a uniform policy $\pi^{\star}$ on state $z_{1}$. Formally, $\mathbb{P}^{\pi^{\star}}[a_{1}=a\mid z_{1}]=\frac{1}{|A|}$. If both players follow this uniform policy, then for any second step history $z_{2}\in Z_{2}$, it must hold that $\mathbb{P}^{\pi^{\star},\pi^{\star}}[z_{2}]>0$. Recall that the players’ actions are part of $z_{2}$, and for a concrete history of the form $z_{2}=(s_{1},a_{1}^{1},a_{1}^{2},s_{2})$, we have:

[TABLE]

Implying that we can construct a modified policy $\hat{\pi}^{\star}$ by altering $\pi^{\star}$ only for actions in $z_{1}$. Specifically, we increase the probability of selecting action $a_{1}^{1}$ slightly, while decreasing the probability of selecting action $a^{2}_{1}$ accordingly. If the change is sufficiently small, then there exists such a policy $\hat{\pi}^{\star}$ for which the following strict inequality holds:

[TABLE]

Now, fix a particular history $z_{2}^{\prime}\in Z_{2}$, and define $Z^{\prime}\coloneqq Z\setminus\{z_{2}^{\prime}\}$, we rearrange Equation 8 and get:

[TABLE]

Notice that only the left-hand side of the equation depends on $z_{2}^{\prime}$. In other words, since our policies are history-dependent, we can modify $\pi^{1}$ and $\pi^{2}$ conditioned on $z_{2}^{\prime}$ without affecting the right-hand side.

We leverage this observation in conjunction with selecting the uniform policy solely at $z_{1}$ as follows:

[TABLE]

for any policies $\pi^{1},\pi^{2},\pi^{3},\pi^{4}$, from which we obtain

[TABLE]

This shows that the sum of the value functions is constant for any pair of policies at each second-stage history $z_{2}^{\prime}\in Z_{2}$.

Now, instead of using the uniform policy at $z_{1}$, we apply the modified policy described in Equation 9 to ensure that the difference in visitation probabilities is nonzero. This, in turn, implies that for any pair of policies, the following holds:

[TABLE]

By combining the observation that $V^{\pi^{1},\pi^{2}}(z_{2}^{\prime})=V^{\pi^{2},\pi^{1}}(z_{2}^{\prime})$ for all $z_{2}^{\prime}\in Z_{2}$, with the fact that the sum of these values is constant across all policy pairs, we conclude:

[TABLE]

completing our proof. ∎

Appendix G Online linear optimization

Online linear optimization is a sequential game between a learner and the environment played for $T$ rounds. Before the game begins, the environment privately generates a sequence of linear loss functions $l_{1},\dots,l_{T}:U\mapsto\mathbb{R}$ where $U\subseteq\mathbb{R}^{m}$ is a compact convex set. At every round $t$, the learner predicts $u_{t}\in U$. Only then, the environment reveals $l_{t}$ and the learner suffers $l_{t}(u_{t})$. The regret of the learner is defined as

[TABLE]

A learning algorithm is said to be consistent if its regret grows sub-linearly with time, i.e., $R(T)\leq o(T)$. This ensures that, on average, the algorithm performs asymptotically as well as the best fixed strategy in hindsight.

Online Gradient Descent (OGD).

OGD refers to the online optimization algorithm that starts from some $u_{1}\in U$, then on every round $t$ updates: (1) $u_{t+1}^{\prime}=u_{t}-\eta\,l_{t}$ where $\eta>0$ is the learning rate; (2) sets $u_{t+1}$ to be the Euclidean projection of $u_{t+1}^{\prime}$ onto $U$. For a specific choice of $\eta$, OGD achieves regret bounded by $O(GD\sqrt{T})$ when $\|l_{t}\|\leq G$ for all $t=1,\dots,T$ and $\max_{u,u^{\prime}\in U}\|u-u^{\prime}\|\leq D$ (see, e.g., (Hazan, 2022)). Note that OGD operates in time complexity of $O(T(m+P))$, where $P$ is the time to compute the projection on $U$. Thus if $P$ is polynomial in the problem parameters, the OGD runs in polynomial time as well.