Human-in-the-Loop Testing of AI Agents for Air Traffic Control with a Regulated Assessment Framework

TL;DR

This paper introduces a human-in-the-loop evaluation framework for AI agents in Air Traffic Control, utilizing a regulator-certified simulator to ensure realistic assessment aligned with real-world standards.

Contribution

It presents a novel, regulator-based assessment framework involving human experts, bridging the gap between academic models and real-world operational environments.

Findings

Framework enables authentic performance measurement

Involves expert human instructors in evaluation

Aligns AI assessment with real-world standards

Abstract

We present a rigorous, human-in-the-loop evaluation framework for assessing the performance of AI agents on the task of Air Traffic Control, grounded in a regulator-certified simulator-based curriculum used for training and testing real-world trainee controllers. By leveraging legally regulated assessments and involving expert human instructors in the evaluation process, our framework enables a more authentic and domain-accurate measurement of AI performance. This work addresses a critical gap in the existing literature: the frequent misalignment between academic representations of Air Traffic Control and the complexities of the actual operational environment. It also lays the foundations for effective future human-machine teaming paradigms by aligning machine performance with human assessment targets.