Trusted Knowledge Extraction for Operations and Maintenance Intelligence

TL;DR

This paper evaluates NLP and LLM tools for extracting operational and maintenance knowledge from confidential aviation data, highlighting challenges and proposing solutions for trusted applications in safety-critical industries.

Contribution

It provides a comprehensive assessment of NLP and LLM capabilities for knowledge extraction in aviation maintenance, including a curated dataset and trust-related insights.

Findings

Significant performance limitations of current NLP and LLM tools in confidential, mission-critical contexts.

Challenges in ensuring trustworthiness and technical readiness of NLP/LLM for aviation use cases.

Open-source dataset provided for benchmarking and further research.

Abstract

Deriving operational intelligence from organizational data repositories is a key challenge due to the dichotomy of data confidentiality vs data integration objectives, as well as the limitations of Natural Language Processing (NLP) tools relative to the specific knowledge structure of domains such as operations and maintenance. In this work, we discuss Knowledge Graph construction and break down the Knowledge Extraction process into its Named Entity Recognition, Coreference Resolution, Named Entity Linking, and Relation Extraction functional components. We then evaluate sixteen NLP tools in concert with or in comparison to the rapidly advancing capabilities of Large Language Models (LLMs). We focus on the operational and maintenance intelligence use case for trusted applications in the aircraft industry. A baseline dataset is derived from a rich public domain US Federal Aviation Administration dataset focused on equipment failures or maintenance requirements. We assess the zero-shot performance of NLP and LLM tools that can be operated within a controlled, confidential environment (no data is sent to third parties). Based on our observation of significant performance limitations, we discuss the challenges related to trusted NLP and LLM tools as well as their Technical Readiness Level for wider use in mission-critical industries such as aviation. We conclude with recommendations to enhance trust and provide our open-source curated dataset to support further baseline testing and evaluation.