LLM-based Zero-shot Triple Extraction for Automated Ontology Generation from Software Engineering Standards

TL;DR

This paper presents an LLM-assisted approach for relation triple extraction from software engineering standards to automate ontology generation, demonstrating comparable or superior performance to existing methods.

Contribution

It introduces a novel LLM-based workflow for automated ontology generation from unstructured software engineering standards texts.

Findings

LLM-assisted method achieves high accuracy in relation triple extraction.

The approach outperforms traditional OpenIE methods in certain metrics.

The workflow effectively handles noisy, domain-specific text for ontology creation.

Abstract

Ontologies have supported knowledge representation and white-box reasoning for decades; thus, the automated ontology generation (AOG) plays a crucial role in scaling their use. Software engineering standards (SES) consist of long, unstructured text (with high noise) and paragraphs with domain-specific terms. In this setting, relation triple extraction (RTE), together with term extraction, constitutes the first stage toward AOG. This work proposes an open-source large language model (LLM)-assisted approach to RTE for SES. Instead of solely relying on prompt-engineering-based methods, this study promotes the use of LLMs as an aid in constructing ontologies and explores an effective AOG workflow that includes document segmentation, candidate term mining, LLM-based relation inference, term normalization, and cross-section alignment. Expert-annotated reference sets at three granularities are constructed and used to evaluate the ontology generated from the study. The results show that it is comparable and potentially superior to the OpenIE method of triple extraction.

Figures

Tables

Table 1. TABLE I : Counts for Pred (system outputs) and expert-annotated reference sets (Short/Medium/Long).

Dataset	#Nodes	#Triples	#Islands
Pred-LLM (average)	55	52	12
Pred-openIE	53	74	4
Ref–Short	68	83	6
Ref–Medium	72	54	26
Ref–Long	32	25	8

Full text

I Introduction

Automated ontology learning has been discussed since 2007 [5], but only in recent years has research on automated ontology generation (AOG) emerged [2, 20, 6, 15]. Traditionally, ontologies are created manually [22], semi-automatically from text [18], or from structured/tabular data such as relational databases [14]. Recent advancements in automated ontology generation leverage deep learning methods [20, 2]. Originally text-centric in natural language processing (NLP), LLMs now function as general-purpose models that handle multimodal inputs [11, 27] and thus are selected to serve as the main assistance for this AOG study.

It is evident that LLMs have been able to help ontology generation from text [4, 13, 21]. When prompted with a passage, an LLM can be instructed to output ontology artifacts in a designated schema (e.g., triples in JSON or OWL/Turtle). However, naive prompting for AOG faces three challenges: 1) Reproducibility and prompt sensitivity: sometimes, when the prompt changes or there is no standard prompt format, the generated content can vary. Even using the same prompt, the generated result (e.g., terms and relations) will vary. 2) Scaling to long or multimodal sources: the generation involving processing a large corpus of text, or multimodal information(such as text and images) requires cross-section term consolidation. A large corpus of text will include many chapters and paragraphs; thus, ontology alignment will need to be considered so that the terms will be able to match the terms in another section. 3) Engineering workflow. When the problem size is large and requires ongoing maintenance and evolution, a mature workflow is needed for the complex task of AOG.

Software engineering standards (SES) may have been implicitly encoded in LLM in the form of model parameters during pretraining; however, to the best of our knowledge, the effort of explicitly representing and extracting that knowledge in a structured way, such as in an ontology, which supports knowledge reasoning [26, 25] and human-centered/human-in-the-loop knowledge representation [23, 24], has not been thoroughly investigated. Due to the complexity involved in the tasks of AOG for SES, the problem of this study is deliberately scoped as follows: 1) A single SES and its official short version are used to keep the text focused and easier to build evaluation benchmarks. 2) It is concentrated on relation triple extraction (RTE) as the first stage toward an LLM-based, fully automated ontology-generation pipeline whose goals are usability, faithfulness to the source text, and maximal coverage. 3) It employs open-source LLMs (e.g., Mistral-7B) and encoder-only models (e.g., BERT) due to cost control and potential privacy concerns [12] in the usage of such methods.

Compared with prevalent TBox-prioritized methodologies (first identifying types and taxonomy) for ontology creation [4, 13, 21], this study proposes an assertion-led, ABox (instance-level assertion)–TBox co-extraction AOG approach for processing textual SES, designs an engineering workflow and best practices for utilizing open-source LLM, and explores metrics for article data sets that are noisy and have open relations. The subsequent sections are organized as follows: Section II reviews related work. Section III details the methodology. Section IV presents experiments and preliminary results. Section V discusses limitations and future work and concludes.

II Related Work

Giglou et al. [4] present LLMs4OL, a conceptual framework that spans tasks across TBox and ABox for ontology learning with LLMs. In their experimental emphasis—and in Lo et al. [17]—the approach is TBox-first, starting with type recognition and taxonomy induction that is limited primarily to is-a hierarchies. In contrast, this study emphasize extracting entity–relation assertions (triples) from SES text, and future work will focus on deriving concepts and their hierarchies from these assertions. Prior evaluations have been conducted on general domain resources (e.g., WordNet), geographic resources (e.g., GeoNames), and biomedical resources (e.g., NCI), and assess a mix of commercial and open-source LLMs. Our work targets SES—an open-relation setting with higher noise and scarce gold ontologies—and builds an engineering workflow exclusively with open-source LLMs. Related work by Lippolis et al. [16] focuses on prompt engineering (PE), proposing two improved prompting techniques for ontology generation.

This work [13] works on a semi-automated workflow for knowledge graph construction, assisted by LLMs (such as ChatGPT 3.5). Human-in-the-loop is involved in different stages of their LLM-supported pipeline. Their approach relies on collecting competency questions to create an ontology and to fill the data in the ontology. They apply their methodology to creating an ontology and KGs about deep learning (DL) methodologies extracted from scholarly publications in the biodiversity domain.

Gong et al. [7] propose ZS-SKA, an approach for prompt-based zero-shot RTE, and apply semantic knowledge augmentation to recognize unseen relations. Their data augmentation is through translating a sentence from its original seen relation to a new unseen relation using an analogy. In the word level, they use top-10 similar words as candidates for new words. Like our work, theirs operates in a zero-shot setting; yet, their datasets (e.g., FewRel) for evaluation are sentence-level with predefined relation sets [8]. In contrast, our study targets SES with open-set relations, article-level with multiple paragraphs and higher noise, and focuses on extracting entity–relation triples.

III Methodology

III-A Workflow Overview

Compared with TBox-prioritized methods (first identifying types and taxonomy), this study propose an assertion-led, ABox–TBox coextraction pipeline: from SES text, relation triplet extraction (RTE) yields instance-level assertions and schema-level candidates (e.g., classes, is-a, domain/range), which could be later validated and consolidated into an OWL ontology. The workflow shown in Figure 1 consists of two main phases, the upper components of which focus on the subtasks of ontology generation using LLMs. The lower components are for postprocessing, including result combination and various checks, which are planned for the next step.

The subtasks shown in the green rectangle are executed in parallel, allowing some processes to run simultaneously. Those generations can be triples or relations, or even fixing the quality issues according to the content checking results.

III-B Standard Selection for this Study

Among many available software engineering standards, the ”Software Engineering Code of Ethics and Professional Practice” (SECEPP) is selected for initial exploration. The main reasons for choosing this document include:

1. It is a publicly available software engineering standard and ”is meant to be a useful code, a document that can inform practice and education” [1].
2. It is concise and not too long, but long enough to be used for starting the investigation.
3. It has a short version (278 words) and a full version (2653 words), where the short version can be used to do concept verification, and the full version is suitable for testing and further investigation. The organization of the chapters and lists imports common noise and challenges of SES processing, which facilitate the migration of this method to work on other types of documents.
4. For automated software engineering and evolution, it is essential that those agents comply with the ethics provided in this article. Thus, it is significant to have it in ontology form/ structured form.

III-C Building an Ontology Scaffold

One challenge of prompt engineering for open-source LLMs is to generate a stable format of the output. The performance of the LLMs could be totally different for different required output formats (e.g., a simple array or JSON triplet format). It may even decide whether the whole route will be successful or not. Thus, instead of relying on LLMs to generate everything, this study primarily relies on the capability of LLMs to find the relations of nodes (terms). As a result, a concept–relation graph (triples) is automatically extracted from the code text. This graph serves as an ontology scaffold for subsequent enrichment.

Algorithm 1 in pseudo code describes the process of building an ontology scaffold $G=(V,E)$ from SECEPP by processing text sentence-by-sentence. For each sentence $S$, spaCy [9] extracts noun phrases and verbs to form candidate entities $P$ and relation vocabulary $V_{b}$; both are cleaned and deduplicated. If no verb candidates exist, the sentence is saved as an orphan $O$. A constrained prompt is composed with $(S,P,V_{b})$; a low-temperature LLM returns JSON triples $T$ under a dynamic token budget (calculated according to the number of terms of the sentence). The method strictly parses the JSON (retry up to $k=3$), normalizes each term (e.g., engineers to engineer, running to run), adds nodes to $V$, and edges to $E$. The lightweight term normalization is via spaCy lemmatization (e.g., engineers → engineer), preserving acronyms/proper nouns and rejecting overly aggressive changes using a similarity threshold. After processing all sentences, remaining orphans are inserted into $V$. The result is a low-noise scaffold ready for later typing/merging. Figure 2 panel (d) is an exemplified scaffold generated by the automated workflow of this study.

III-D Data and Expert-annotated Reference Sets

Three expert-annotated reference sets—Ref-Long, Ref-Medium, and Ref-Short—are constructed manually for the short version of the SECEPP. They enable us to evaluate the robustness of the method across various term granularities and levels of strictness, and their visualization corresponds to Figure 2 panels (a), (b), and (c). Ref-Short preserves fine-grained clauses (higher density, recall-oriented as recall is more difficult to achieve and thus more thoroughly tested), Ref-Medium merges minor edges of Ref-Short, and Ref-Long preserves longer clauses and core relations (precision-oriented).

Table I includes the numbers of Nodes, Triples, and Islands for the three reference sets. As expected, Ref-Short has the largest graph (68 nodes/83 triples), with fewer islands. Ref-Medium is more fragmented (26 islands) due to pruning connections, providing a balanced middle ground. Ref-Long is the most compact (32 nodes/25 triples).

IV Experiments and Preliminary Results

IV-A Systems Compared

Pred-LLM is the result from this research–Figure 2 panel (d) is one representative output out of five runs, the mean performance of which is reported in Figure 3. An open-source 7B LLM (Mistral-7B-Instruct-v0.1 [10]) is used in this study of open-set relation generation. Code and data are available at https://github.com/songhui01/aog-ses.git.

Pred-openIE is a sentence-level OpenIE baseline–an example is shown in Figure 2 panel (e). Stanford OpenIE [3] implemented in Stanford CoreNLP [19] is employed.

IV-B Evaluation Protocol

Fully automated outputs (no post-editing) are evaluated against three ref-standard reference sets for the short SECEPP: Ref-Short, Ref-Medium, Ref-Long (Figure  2; stats in Table I).

Quality is assessed at the node and triple levels via an embedding-similarity threshold sweep $\tau\in\{0.10,0.15,\dots,0.90\}$; Precision, Recall and F1 are reported as curves (Figure 3, top: nodes; bottom: triples; columns: Short/Medium/Long). Matches require the similarity of the concatenation of subject, relation, and object (a single space delimiteris used) to pass $\tau$ (greedy one-to-one alignment).

IV-C Key Findings

(1) Recall is highest on Ref-Long and lowest on Ref-Short, confirming the recall-stress hypothesis for dense reference sets. The Ref-Long is a compact backbone, and it eases recall (fewer edges), but any extra predicted edges not in the backbone are counted as false positives, lowering precision.

(2) Pred-LLM often yields higher precision than OpenIE across $\tau$, and only when $\tau\!>\!0.5$, the recall gap narrows, which makes sense as Pred-LLM’s result has a much smaller number than Pred-openIE’s result ($42$ vs. $74$ under Ref-Short) and spurious/duplicate triples from OpenIE are filtered.

(3) An additional observation from the experimental checks is that when $\tau$ reaches around 0.6, the triples generally stabilize and become reliable. The supporting evidence could not be included in this short paper, but this preliminary finding motivates further investigation of the optimal $\tau$ value as the key indicator for this type of study.

(4 )A 7B open-source LLM is already competitive with OpenIE—higher precision on both nodes and triples, and comparable recall on stricter reference sets (when $\tau\!>\!0.5$)—while recall on Ref-Short remains the main headroom to be improved.

(5) Triple matching is strictly harder than node matching: errors in any of head/relation/tail cause a miss, so generally both precision and recall drop at the triple level.

IV-D Discussion

The results suggest that a 7B open-source LLM can already be competitive with OpenIE in precision for node and triple extraction. This indicates that stronger LLMs—either with larger parameter sizes or fine-tuned on domain-specific data—may further improve extraction quality.

However, this preliminary study has several limitations. First, it evaluates the workflow using a single LLM and thus lacks comparative experiments across LLM families and scales, as well as across backbone variants such as dense vs. MoE. Second, the reference datasets are manually constructed by a single annotator (domain expert) at three different granularities to support systematic evaluation in the absence of existing gold sets for this domain/task; consequently, inter-annotator agreement (IAA) is not reported. These labels are intended for formative evaluation rather than a community benchmark. In future work, we will include multiple annotators and report IAA to improve label reliability and support the construction of a public benchmark dataset.

V Conclusion / Future Work

This work initiates LLM-assisted ontology generation for software engineering standards, proposes a fully automated workflow, constructs three expert-annotated reference sets (single annotator in this priliminary study) for the short version, and evaluates with threshold-sweep metrics at node and triple levels. Preliminary results show that a 7B open-source LLM is competitive with a Stanford OpenIE baseline (often higher precision, comparable recall on stricter reference sets).

Next, we will work on the improvement of the results, extend from the short to the full (long) version and to additional SES documents; improve recall via cross-sentence (coreference) handling, and small-scope fine-tuning; and perform the later ontology-transformation phases to produce and release a formal OWL 2 ontology and public benchmarks.

VI Acknowledgements

This material is based upon work supported by the National Science Foundation under Grant No. 2451397.

AI assistants (OpenAI’s GPT-4.x and Google’s Gemini suggestions embedded in Google Colab) were used for rapid prototyping during the experiments. No AI was used to fabricate data or results. The code was comprehensively reviewed and revised by the author. AI tools were also used to rephrase some passages for conciseness and grammatical correctness.