Mastering the Digital Art of War: Developing Intelligent Combat Simulation Agents for Wargaming Using Hierarchical Reinforcement Learning

TL;DR

This paper introduces a hierarchical reinforcement learning framework combined with multi-model and hybrid AI approaches to develop intelligent combat simulation agents, aiming to improve realism and efficiency in military wargaming scenarios.

Contribution

It proposes a novel HRL-based architecture with targeted observation abstractions and hybrid AI integration, advancing the development of adaptable and efficient combat simulation agents.

Findings

Localized observation abstraction improves computational efficiency.

Multi-model framework enhances performance with diverse AI methods.

Initial tests showed no performance gain but provided valuable insights.

Abstract

In today's rapidly evolving military landscape, advancing artificial intelligence (AI) in support of wargaming becomes essential. Despite reinforcement learning (RL) showing promise for developing intelligent agents, conventional RL faces limitations in handling the complexity inherent in combat simulations. This dissertation proposes a comprehensive approach, including targeted observation abstractions, multi-model integration, a hybrid AI framework, and an overarching hierarchical reinforcement learning (HRL) framework. Our localized observation abstraction using piecewise linear spatial decay simplifies the RL problem, enhancing computational efficiency and demonstrating superior efficacy over traditional global observation methods. Our multi-model framework combines various AI methodologies, optimizing performance while still enabling the use of diverse, specialized individual behavior models. Our hybrid AI framework synergizes RL with scripted agents, leveraging RL for high-level decisions and scripted agents for lower-level tasks, enhancing adaptability, reliability, and performance. Our HRL architecture and training framework decomposes complex problems into manageable subproblems, aligning with military decision-making structures. Although initial tests did not show improved performance, insights were gained to improve future iterations. This study underscores AI's potential to revolutionize wargaming, emphasizing the need for continued research in this domain.