Bidding Games on Markov Decision Processes with Quantitative Reachability Objectives

TL;DR

This paper introduces a new class of bidding games on Markov decision processes where two players bid for control over actions to influence reachability probabilities, extending traditional bidding game analysis to stochastic environments.

Contribution

It formalizes bidding games on MDPs with quantitative reachability objectives and develops algorithms to approximate thresholds and optimal strategies, including exact solutions for acyclic MDPs.

Findings

Developed value-iteration algorithms for general MDPs

Computed exact solutions for acyclic MDPs

Showed threshold determination is as hard as simple-stochastic games

Abstract

Graph games are fundamental in strategic reasoning of multi-agent systems and their environments. We study a new family of graph games which combine stochastic environmental uncertainties and auction-based interactions among the agents, formalized as bidding games on (finite) Markov decision processes (MDP). Normally, on MDPs, a single decision-maker chooses a sequence of actions, producing a probability distribution over infinite paths. In bidding games on MDPs, two players -- called the reachability and safety players -- bid for the privilege of choosing the next action at each step. The reachability player's goal is to maximize the probability of reaching a target vertex, whereas the safety player's goal is to minimize it. These games generalize traditional bidding games on graphs, and the existing analysis techniques do not extend. For instance, the central property of traditional bidding games is the existence of a threshold budget, which is a necessary and sufficient budget to guarantee winning for the reachability player. For MDPs, the threshold becomes a relation between the budgets and probabilities of reaching the target. We devise value-iteration algorithms that approximate thresholds and optimal policies for general MDPs, and compute the exact solutions for acyclic MDPs, and show that finding thresholds is at least as hard as solving simple-stochastic games.