Integrating Large Language Models and Knowledge Graphs for Extraction and Validation of Textual Test Data

TL;DR

This paper presents a hybrid approach combining Knowledge Graphs and Large Language Models to automate extraction and validation of complex aerospace test data, improving efficiency and enabling cross-report analysis.

Contribution

It introduces a novel methodology integrating KGs and LLMs for data extraction and validation in aerospace manufacturing, extending ontologies and benchmarking LLM performance.

Findings

Effective data extraction and validation using LLMs demonstrated.

Benchmarking shows varying LLM performance on aerospace data.

Automation reduces manual effort and enhances data consistency.

Abstract

Aerospace manufacturing companies, such as Thales Alenia Space, design, develop, integrate, verify, and validate products characterized by high complexity and low volume. They carefully document all phases for each product but analyses across products are challenging due to the heterogeneity and unstructured nature of the data in documents. In this paper, we propose a hybrid methodology that leverages Knowledge Graphs (KGs) in conjunction with Large Language Models (LLMs) to extract and validate data contained in these documents. We consider a case study focused on test data related to electronic boards for satellites. To do so, we extend the Semantic Sensor Network ontology. We store the metadata of the reports in a KG, while the actual test results are stored in parquet accessible via a Virtual Knowledge Graph. The validation process is managed using an LLM-based approach. We also conduct a benchmarking study to evaluate the performance of state-of-the-art LLMs in executing this task. Finally, we analyze the costs and benefits of automating preexisting processes of manual data extraction and validation for subsequent cross-report analyses.