Rethinking Priority Scheduling for Sequential Multi-Agent Decision Making in Stackelberg Games

TL;DR

This paper analyzes how the order of decision-making in N-level Stackelberg Games affects equilibrium points and introduces the Hierarchical Priority Adjustment method to optimize agent decision order, improving multi-agent control performance.

Contribution

The paper provides a formal analysis of decision order effects in Stackelberg Games and proposes a dynamic priority adjustment method for better multi-agent decision making.

Findings

HPA outperforms benchmark algorithms in high-precision control tasks.

Adjusting agent decision order significantly impacts equilibrium stability.

The method adapts robustly to changing environments.

Abstract

Current research applying N-level Stackelberg Game to multi-agent systems often uses the default decision order of agents provided by the environment. However, this raises the question: does the order of agents necessarily affect the final equilibrium point of the game? To address this, we formally analyze the N-level Stackelberg Game, where changing the order in which agents make decisions typically leads to an overdetermined system. As a result, the equilibrium point shifts unless special structural conditions are satisfied. Based on this analysis, we propose the Hierarchical Priority Adjustment (HPA) method, which adjusts and selects the agents' decision order. At the upper level, an upper policy dynamically selects the optimal decision order of agents based on the current game state. At the lower level, agents execute strategies in the Spatio-Temporal Sequential Markov Game (STMG) according to the selected order. To coordinate learning across time scales, we employ a slow-fast update scheme with shared intrinsic rewards derived from the advantage function of the upper policy. Experimental results on high-precision control tasks, including multi-agent MuJoCo, show that HPA outperforms benchmark algorithms and robustly adapts to changing environments. These results highlight the crucial role of optimizing the agents' decision order in N-level Stackelberg Game.