GOFAI meets Generative AI: Development of Expert Systems by means of Large Language Models

TL;DR

This paper presents a hybrid approach combining large language models with symbolic reasoning to develop expert systems that are interpretable, reliable, and verifiable, addressing issues like hallucinations in LLMs.

Contribution

It introduces a controlled, prompt-based method to extract and validate symbolic knowledge from LLMs, enhancing transparency and trustworthiness of expert systems.

Findings

Strong adherence to facts in generated knowledge bases

High semantic coherence demonstrated in experiments

Hybrid approach improves reliability of AI in sensitive domains

Abstract

The development of large language models (LLMs) has successfully transformed knowledge-based systems such as open domain question nswering, which can automatically produce vast amounts of seemingly coherent information. Yet, those models have several disadvantages like hallucinations or confident generation of incorrect or unverifiable facts. In this paper, we introduce a new approach to the development of expert systems using LLMs in a controlled and transparent way. By limiting the domain and employing a well-structured prompt-based extraction approach, we produce a symbolic representation of knowledge in Prolog, which can be validated and corrected by human experts. This approach also guarantees interpretability, scalability and reliability of the developed expert systems. Via quantitative and qualitative experiments with Claude Sonnet 3.7 and GPT-4.1, we show strong adherence to facts and semantic coherence on our generated knowledge bases. We present a transparent hybrid solution that combines the recall capacity of LLMs with the precision of symbolic systems, thereby laying the foundation for dependable AI applications in sensitive domains.