# Playing Markov Games Without Observing Payoffs

## TL;DR

This paper extends the ability to learn and compete in symmetric Markov games without payoff observations, using only opponent actions and known transition dynamics, by reducing the problem to online learning.

## Contribution

It introduces a formal framework for zero-sum symmetric Markov games with limited information and provides algorithms that achieve asymptotic performance guarantees.

## Key findings

- Players can compete without payoff observations using only opponent actions.
- Algorithms run in polynomial time relative to game size and episodes.
- The approach applies to both matrix and Markov games, broadening robust learning scenarios.

## Abstract

Optimization under uncertainty is a fundamental problem in learning and decision-making, particularly in multi-agent systems. Previously, Feldman, Kalai, and Tennenholtz [2010] demonstrated the ability to efficiently compete in repeated symmetric two-player matrix games without observing payoffs, as long as the opponents actions are observed. In this paper, we introduce and formalize a new class of zero-sum symmetric Markov games, which extends the notion of symmetry from matrix games to the Markovian setting. We show that even without observing payoffs, a player who knows the transition dynamics and observes only the opponents sequence of actions can still compete against an adversary who may have complete knowledge of the game. We formalize three distinct notions of symmetry in this setting and show that, under these conditions, the learning problem can be reduced to an instance of online learning, enabling the player to asymptotically match the return of the opponent despite lacking payoff observations. Our algorithms apply to both matrix and Markov games, and run in polynomial time with respect to the size of the game and the number of episodes. Our work broadens the class of games in which robust learning is possible under severe informational disadvantage and deepens the connection between online learning and adversarial game theory.