Training Environment for High Performance Reinforcement Learning

TL;DR

Tunnel is an open-source reinforcement learning environment simulating high-performance aircraft, integrating realistic flight dynamics and customizable features to enhance research and operational training in autonomous air combat.

Contribution

The paper introduces Tunnel, a flexible, open-source RL training environment with realistic aircraft physics, enabling rapid customization for research and operational needs.

Findings

Demonstrated a week-long trade study on training methods and threat scenarios.

Enabled rapid customization of training environments in days instead of months.

Facilitated collaboration between researchers and mission planners for military advantage.

Abstract

This paper presents Tunnel, a simple, open source, reinforcement learning training environment for high performance aircraft. It integrates the F16 3D nonlinear flight dynamics into OpenAI Gymnasium python package. The template includes primitives for boundaries, targets, adversaries and sensing capabilities that may vary depending on operational need. This offers mission planners a means to rapidly respond to evolving environments, sensor capabilities and adversaries for autonomous air combat aircraft. It offers researchers access to operationally relevant aircraft physics. Tunnel code base is accessible to anyone familiar with Gymnasium and/or those with basic python skills. This paper includes a demonstration of a week long trade study that investigated a variety of training methods, observation spaces, and threat presentations. This enables increased collaboration between researchers and mission planners which can translate to a national military advantage. As warfare becomes increasingly reliant upon automation, software agility will correlate with decision advantages. Airmen must have tools to adapt to adversaries in this context. It may take months for researchers to develop skills to customize observation, actions, tasks and training methodologies in air combat simulators. In Tunnel, this can be done in a matter of days.