Transforming Expert Knowledge into Scalable Ontology via Large Language Models

TL;DR

This paper presents a novel framework that leverages large language models combined with expert input and iterative prompt tuning to automate and scale taxonomy alignment with high accuracy, surpassing human performance.

Contribution

It introduces an innovative approach integrating LLMs, expert calibration, and prompt optimization for scalable, accurate taxonomy alignment in domain-specific applications.

Findings

Achieved an F1-score of 0.97 on a concept essentiality mapping task.

Outperformed the human benchmark of 0.68 in taxonomy alignment accuracy.

Demonstrated effective scaling of taxonomy alignment with high-quality results.

Abstract

Having a unified, coherent taxonomy is essential for effective knowledge representation in domain-specific applications as diverse terminologies need to be mapped to underlying concepts. Traditional manual approaches to taxonomy alignment rely on expert review of concept pairs, but this becomes prohibitively expensive and time-consuming at scale, while subjective interpretations often lead to expert disagreements. Existing automated methods for taxonomy alignment have shown promise but face limitations in handling nuanced semantic relationships and maintaining consistency across different domains. These approaches often struggle with context-dependent concept mappings and lack transparent reasoning processes. We propose a novel framework that combines large language models (LLMs) with expert calibration and iterative prompt optimization to automate taxonomy alignment. Our method integrates expert-labeled examples, multi-stage prompt engineering, and human validation to guide LLMs in generating both taxonomy linkages and supporting rationales. In evaluating our framework on a domain-specific mapping task of concept essentiality, we achieved an F1-score of 0.97, substantially exceeding the human benchmark of 0.68. These results demonstrate the effectiveness of our approach in scaling taxonomy alignment while maintaining high-quality mappings and preserving expert oversight for ambiguous cases.