Maneuver Decision-Making Through Automatic Curriculum Reinforcement Learning Without Handcrafted Reward functions

TL;DR

This paper introduces an automatic curriculum reinforcement learning approach for unmanned aerial vehicle maneuver decision-making, enabling effective autonomous air combat strategies without handcrafted reward functions.

Contribution

The paper presents a novel automatic curriculum learning method that trains agents to make maneuver decisions from scratch without manually designed reward functions.

Findings

Agents learn effective maneuvering strategies in air combat scenarios.

Automatic curriculum learning is essential for successful reinforcement learning in this context.

Trained agents can perform tracking, attacking, and escaping maneuvers effectively.

Abstract

Maneuver decision-making is the core of unmanned combat aerial vehicle for autonomous air combat. To solve this problem, we propose an automatic curriculum reinforcement learning method, which enables agents to learn effective decisions in air combat from scratch. The range of initial states are used for distinguishing curricula of different difficulty levels, thereby maneuver decision is divided into a series of sub-tasks from easy to difficult, and test results are used to change sub-tasks. As sub-tasks change, agents gradually learn to complete a series of sub-tasks from easy to difficult, enabling them to make effective maneuvering decisions to cope with various states without the need to spend effort designing reward functions. The ablation studied show that the automatic curriculum learning proposed in this article is an essential component for training through reinforcement learning, namely, agents cannot complete effective decisions without curriculum learning. Simulation experiments show that, after training, agents are able to make effective decisions given different states, including tracking, attacking and escaping, which are both rational and interpretable.