Light Aircraft Game : Basic Implementation and training results analysis

TL;DR

This paper explores multi-agent reinforcement learning in a combat environment called LAG, comparing hierarchical on-policy and off-policy algorithms to understand their training stability and coordination capabilities.

Contribution

It introduces a detailed combat environment setup and evaluates two algorithms, HAPPO and HASAC, highlighting their strengths in different scenarios.

Findings

HASAC performs well in simple coordination tasks

HAPPO shows better adaptability in complex missile combat

Insights into on-policy vs off-policy trade-offs in MARL

Abstract

This paper investigates multi-agent reinforcement learning (MARL) in a partially observable, cooperative-competitive combat environment known as LAG. We describe the environment's setup, including agent actions, hierarchical controls, and reward design across different combat modes such as No Weapon and ShootMissile. Two representative algorithms are evaluated: HAPPO, an on-policy hierarchical variant of PPO, and HASAC, an off-policy method based on soft actor-critic. We analyze their training stability, reward progression, and inter-agent coordination capabilities. Experimental results show that HASAC performs well in simpler coordination tasks without weapons, while HAPPO demonstrates stronger adaptability in more dynamic and expressive scenarios involving missile combat. These findings provide insights into the trade-offs between on-policy and off-policy methods in multi-agent settings.