HGFormer: A Hierarchical Graph Transformer Framework for Two-Stage Colonel Blotto Games via Reinforcement Learning

TL;DR

This paper introduces HGformer, a hierarchical graph Transformer framework that enhances resource allocation strategies in two-stage Colonel Blotto games through reinforcement learning, addressing complex constraints and sequential dependencies.

Contribution

The paper presents a novel hierarchical graph Transformer model with a layer-by-layer reinforcement learning algorithm for two-stage resource allocation games, improving efficiency and performance.

Findings

HGformer outperforms existing methods in resource allocation efficiency.

The approach achieves higher adversarial payoffs in complex dynamic scenarios.

Experimental results validate the effectiveness of the hierarchical and reinforcement learning design.

Abstract

Two-stage Colonel Blotto game represents a typical adversarial resource allocation problem, in which two opposing agents sequentially allocate resources in a network topology across two phases: an initial resource deployment followed by multiple rounds of dynamic reallocation adjustments. The sequential dependency between game stages and the complex constraints imposed by the graph topology make it difficult for traditional approaches to attain a globally optimal strategy. To address these challenges, we propose a hierarchical graph Transformer framework called HGformer. By incorporating an enhanced graph Transformer encoder with structural biases and a two-agent hierarchical decision model, our approach enables efficient policy generation in large-scale adversarial environments. Moreover, we design a layer-by-layer feedback reinforcement learning algorithm that feeds the long-term returns from lower-level decisions back into the optimization of the higher-level strategy, thus bridging the coordination gap between the two decision-making stages. Experimental results demonstrate that, compared to existing hierarchical decision-making or graph neural network methods, HGformer significantly improves resource allocation efficiency and adversarial payoff, achieving superior overall performance in complex dynamic game scenarios.