LLM Evaluation Based on Aerospace Manufacturing Expertise: Automated Generation and Multi-Model Question Answering

TL;DR

This paper develops specialized evaluation metrics for assessing LLMs in aerospace manufacturing, focusing on their accuracy in answering domain-specific questions and highlighting the need for improved performance.

Contribution

It introduces a novel set of evaluation metrics and question-answering benchmarks tailored for LLMs in aerospace manufacturing, addressing domain-specific accuracy issues.

Findings

LLMs show significant inaccuracies in aerospace domain knowledge.

The proposed evaluation metrics effectively measure LLM performance in specialized fields.

Results indicate a pressing need to enhance LLM capabilities for aerospace applications.

Abstract

Aerospace manufacturing demands exceptionally high precision in technical parameters. The remarkable performance of Large Language Models (LLMs), such as GPT-4 and QWen, in Natural Language Processing has sparked industry interest in their application to tasks including process design, material selection, and tool information retrieval. However, LLMs are prone to generating "hallucinations" in specialized domains, producing inaccurate or false information that poses significant risks to the quality of aerospace products and flight safety. This paper introduces a set of evaluation metrics tailored for LLMs in aerospace manufacturing, aiming to assess their accuracy by analyzing their performance in answering questions grounded in professional knowledge. Firstly, key information is extracted through in-depth textual analysis of classic aerospace manufacturing textbooks and guidelines. Subsequently, utilizing LLM generation techniques, we meticulously construct multiple-choice questions with multiple correct answers of varying difficulty. Following this, different LLM models are employed to answer these questions, and their accuracy is recorded. Experimental results demonstrate that the capabilities of LLMs in aerospace professional knowledge are in urgent need of improvement. This study provides a theoretical foundation and practical guidance for the application of LLMs in aerospace manufacturing, addressing a critical gap in the field.