Hierarchical Reinforcement Learning for Air-to-Air Combat

TL;DR

This paper presents a hierarchical reinforcement learning approach with expert knowledge integration for AI-controlled air-to-air combat, demonstrating competitive performance in DARPA's AlphaDogfight Trials.

Contribution

It introduces a novel hierarchical RL architecture with reward shaping and modular policies, achieving high performance in simulated combat scenarios.

Findings

Achieved 2nd place in DARPA ADT among eight competitors

Defeated a US Air Force F-16 Weapons Instructor Course graduate in match play

Demonstrated effectiveness of hierarchical RL in complex combat simulations

Abstract

Artificial Intelligence (AI) is becoming a critical component in the defense industry, as recently demonstrated by DARPA`s AlphaDogfight Trials (ADT). ADT sought to vet the feasibility of AI algorithms capable of piloting an F-16 in simulated air-to-air combat. As a participant in ADT, Lockheed Martin`s (LM) approach combines a hierarchical architecture with maximum-entropy reinforcement learning (RL), integrates expert knowledge through reward shaping, and supports modularity of policies. This approach achieved a $2^{nd}$ place finish in the final ADT event (among eight total competitors) and defeated a graduate of the US Air Force's (USAF) F-16 Weapons Instructor Course in match play.