Knots: A Large-Scale Multi-Agent Enhanced Expert-Annotated Dataset and LLM Prompt Optimization for NOTAM Semantic Parsing

TL;DR

This paper introduces Knots, a large expert-annotated dataset for NOTAM semantic parsing, and explores prompt optimization techniques for improved multi-agent LLM understanding of complex aviation safety texts.

Contribution

It presents a new high-quality dataset for NOTAM semantic parsing and evaluates prompt-engineering strategies to enhance LLM performance in aviation domain understanding.

Findings

Prompt optimization significantly improves LLM accuracy on NOTAM tasks.

The dataset covers 12,347 NOTAMs across 194 regions, supporting diverse aviation scenarios.

Enhanced models demonstrate better semantic inference for aviation safety analysis.

Abstract

Notice to Air Missions (NOTAMs) serve as a critical channel for disseminating key flight safety information, yet their complex linguistic structures and implicit reasoning pose significant challenges for automated parsing. Existing research mainly focuses on surface-level tasks such as classification and named entity recognition, lacking deep semantic understanding. To address this gap, we propose NOTAM semantic parsing, a task emphasizing semantic inference and the integration of aviation domain knowledge to produce structured, inference-rich outputs. To support this task, we construct Knots (Knowledge and NOTAM Semantics), a high-quality dataset of 12,347 expert-annotated NOTAMs covering 194 Flight Information Regions, enhanced through a multi-agent collaborative framework for comprehensive field discovery. We systematically evaluate a wide range of prompt-engineering strategies and model-adaptation techniques, achieving substantial improvements in aviation text understanding and processing. Our experimental results demonstrate the effectiveness of the proposed approach and offer valuable insights for automated NOTAM analysis systems. Our code is available at: https://github.com/Estrellajer/Knots.