A Study on Effect of Reference Knowledge Choice in Generating Technical Content Relevant to SAPPhIRE Model Using Large Language Model

TL;DR

This paper explores how selecting appropriate reference knowledge influences the accuracy of generating SAPPhIRE model-based technical content using large language models, proposing a retrieval-augmented approach to reduce hallucinations.

Contribution

It introduces a retrieval-augmented generation method to improve technical content accuracy in SAPPhIRE modeling with LLMs, emphasizing the importance of reference knowledge selection.

Findings

Reference knowledge choice significantly impacts content accuracy.

Retrieval-augmented generation reduces hallucinations in LLM outputs.

Method supports building software tools for SAPPhIRE modeling.

Abstract

Representation of systems using the SAPPhIRE model of causality can be an inspirational stimulus in design. However, creating a SAPPhIRE model of a technical or a natural system requires sourcing technical knowledge from multiple technical documents regarding how the system works. This research investigates how to generate technical content accurately relevant to the SAPPhIRE model of causality using a Large Language Model, also called LLM. This paper, which is the first part of the two-part research, presents a method for hallucination suppression using Retrieval Augmented Generating with LLM to generate technical content supported by the scientific information relevant to a SAPPhIRE con-struct. The result from this research shows that the selection of reference knowledge used in providing context to the LLM for generating the technical content is very important. The outcome of this research is used to build a software support tool to generate the SAPPhIRE model of a given technical system.