On Dynamic Programming Theory for Leader-Follower Stochastic Games

TL;DR

This paper develops a dynamic programming framework for leader-follower stochastic games, enabling computation of strong Stackelberg equilibria with scalable algorithms and empirical validation on various benchmarks.

Contribution

It introduces a DP approach over credible sets, proves reduction to MDPs, and provides NP-hardness results and psilon-optimal algorithms for leader strategies.

Findings

Empirical gains in leader value over existing methods.

Scalable algorithms demonstrated on security, resource allocation, and adversarial planning.

Theoretical foundations for DP over credible sets and NP-hardness of optimal leader policy synthesis.

Abstract

Leader-follower general-sum stochastic games (LF-GSSGs) model sequential decision-making under asymmetric commitment, where a leader commits to a policy and a follower best responds, yielding a strong Stackelberg equilibrium (SSE) with leader-favourable tie-breaking. This paper introduces a dynamic programming (DP) framework that applies Bellman recursion over credible sets-state abstractions formally representing all rational follower best responses under partial leader commitments-to compute SSEs. We first prove that any LF-GSSG admits a lossless reduction to a Markov decision process (MDP) over credible sets. We further establish that synthesising an optimal memoryless deterministic leader policy is NP-hard, motivating the development of {\epsilon}-optimal DP algorithms with provable guarantees on leader exploitability. Experiments on standard mixed-motive benchmarks-including security games, resource allocation, and adversarial planning-demonstrate empirical gains in leader value and runtime scalability over state-of-the-art methods.