Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Field

TL;DR

This paper evaluates large language models' ability to identify semantic research topic relationships for automated ontology generation, highlighting that smaller, optimized models can perform comparably to larger ones, thus enabling efficient scientific knowledge structuring.

Contribution

The study provides a comprehensive evaluation of 17 LLMs for research topic relationship identification, introducing a benchmark based on IEEE Thesaurus and demonstrating the effectiveness of smaller, optimized models.

Findings

Several models achieved high F1-scores, with Claude 3 Sonnet reaching 0.967.

Smaller, quantised models can match larger models' performance with proper prompt engineering.

Optimized small models require less computational resources while maintaining accuracy.

Abstract

Ontologies of research topics are crucial for structuring scientific knowledge, enabling scientists to navigate vast amounts of research, and forming the backbone of intelligent systems such as search engines and recommendation systems. However, manual creation of these ontologies is expensive, slow, and often results in outdated and overly general representations. As a solution, researchers have been investigating ways to automate or semi-automate the process of generating these ontologies. This paper offers a comprehensive analysis of the ability of large language models (LLMs) to identify semantic relationships between different research topics, which is a critical step in the development of such ontologies. To this end, we developed a gold standard based on the IEEE Thesaurus to evaluate the task of identifying four types of relationships between pairs of topics: broader, narrower, same-as, and other. Our study evaluates the performance of seventeen LLMs, which differ in scale, accessibility (open vs. proprietary), and model type (full vs. quantised), while also assessing four zero-shot reasoning strategies. Several models have achieved outstanding results, including Mixtral-8x7B, Dolphin-Mistral-7B, and Claude 3 Sonnet, with F1-scores of 0.847, 0.920, and 0.967, respectively. Furthermore, our findings demonstrate that smaller, quantised models, when optimised through prompt engineering, can deliver performance comparable to much larger proprietary models, while requiring significantly fewer computational resources.