Zero-Sum Semi-Markov Games with State-Action-Dependent Discount Factors

TL;DR

This paper studies two-player zero-sum semi-Markov games with state-action-dependent discounting, establishing the existence of value functions and optimal strategies, and providing algorithms for finite cases with convergence guarantees.

Contribution

It introduces a general semi-Markov game model with state-action-dependent discount factors and proves the existence of solutions under regularity conditions, also developing a value iteration algorithm for finite cases.

Findings

Existence of value functions and optimal strategies under certain conditions.

Development of a convergent value iteration algorithm for finite state-action spaces.

Numerical examples demonstrating theoretical results.

Abstract

Semi-Markov model is one of the most general models for stochastic dynamic systems. This paper deals with a two-person zero-sum game for semi-Markov processes. We focus on the expected discounted payoff criterion with state-action-dependent discount factors. The state and action spaces are both Polish spaces, and the payoff function is $\omega$-bounded. We first construct a fairly general model of semi-Markov games under a given semi-Markov kernel and a pair of strategies. Next, based on the standard regularity condition and the continuity-compactness condition for semi-Markov games, we derive a "drift condition" on the semi-Markov kernel and suppose that the discount factors have a positive lower bound, under which the existence of the value function and a pair of optimal stationary strategies of our semi-Markov game are proved by using the Shapley equation. Moreover, when the state and action spaces are both finite, a value iteration-type algorithm for computing the value function and $\varepsilon$-Nash equilibrium of the game is developed. The convergence of the algorithm is also proved. Finally, we conduct numerical examples to demonstrate our main results.