Approachability in Stackelberg Stochastic Games with Vector Costs

TL;DR

This paper extends Blackwell's approachability concept to Stackelberg stochastic games with vector costs, providing a practical strategy and a reinforcement learning algorithm for unknown environments, with theoretical guarantees.

Contribution

It introduces a simple, computationally feasible approachability strategy and a reinforcement learning method for Stackelberg stochastic games with vector costs, including theoretical characterizations.

Findings

Proposes a tractable approachability strategy for Stackelberg stochastic games.

Develops a reinforcement learning algorithm for unknown transition kernels.

Provides a complete characterization of approachability conditions for convex sets.

Abstract

The notion of approachability was introduced by Blackwell [1] in the context of vector-valued repeated games. The famous Blackwell's approachability theorem prescribes a strategy for approachability, i.e., for `steering' the average cost of a given agent towards a given target set, irrespective of the strategies of the other agents. In this paper, motivated by the multi-objective optimization/decision making problems in dynamically changing environments, we address the approachability problem in Stackelberg stochastic games with vector valued cost functions. We make two main contributions. Firstly, we give a simple and computationally tractable strategy for approachability for Stackelberg stochastic games along the lines of Blackwell's. Secondly, we give a reinforcement learning algorithm for learning the approachable strategy when the transition kernel is unknown. We also recover as a by-product Blackwell's necessary and sufficient condition for approachability for convex sets in this set up and thus a complete characterization. We also give sufficient conditions for non-convex sets.