A Hierarchical Deep Reinforcement Learning Framework for 6-DOF UCAV Air-to-Air Combat

TL;DR

This paper introduces a hierarchical deep reinforcement learning framework for 6-DOF UCAV air-to-air combat, dividing decision-making into macro strategy and micro control loops trained with PPO and self-play.

Contribution

It presents a novel hierarchical RL approach with separate loops for strategy and control, using PPO and self-play to enhance combat performance in complex 6-DOF scenarios.

Findings

Inner loop controller outperforms PID in tracking accuracy.

Outer loop strategy achieves higher winning rates through evolving maneuvers.

Hierarchical framework effectively manages complex combat dynamics.

Abstract

Unmanned combat air vehicle (UCAV) combat is a challenging scenario with continuous action space. In this paper, we propose a general hierarchical framework to resolve the within-vision-range (WVR) air-to-air combat problem under 6 dimensions of degree (6-DOF) dynamics. The core idea is to divide the whole decision process into two loops and use reinforcement learning (RL) to solve them separately. The outer loop takes into account the current combat situation and decides the expected macro behavior of the aircraft according to a combat strategy. Then the inner loop tracks the macro behavior with a flight controller by calculating the actual input signals for the aircraft. We design the Markov decision process for both the outer loop strategy and inner loop controller, and train them by proximal policy optimization (PPO) algorithm. For the inner loop controller, we design an effective reward function to accurately track various macro behavior. For the outer loop strategy, we further adopt a fictitious self-play mechanism to improve the combat performance by constantly combating against the historical strategies. Experiment results show that the inner loop controller can achieve better tracking performance than fine-tuned PID controller, and the outer loop strategy can perform complex maneuvers to get higher and higher winning rate, with the generation evolves.