Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domain

TL;DR

This study explores automating domain knowledge extraction for autonomous driving using large language models, highlighting the benefits of human supervision to improve quality and efficiency.

Contribution

It introduces an empirical framework for domain knowledge distillation with LLMs, emphasizing the role of human oversight and providing a web-based assistant tool.

Findings

Automated domain ontology construction is feasible.

Human supervision enhances output quality and efficiency.

Tools for semi-automated knowledge distillation are effective.

Abstract

Engineering knowledge-based (or expert) systems require extensive manual effort and domain knowledge. As Large Language Models (LLMs) are trained using an enormous amount of cross-domain knowledge, it becomes possible to automate such engineering processes. This paper presents an empirical automation and semi-automation framework for domain knowledge distillation using prompt engineering and the LLM ChatGPT. We assess the framework empirically in the autonomous driving domain and present our key observations. In our implementation, we construct the domain knowledge ontology by "chatting" with ChatGPT. The key finding is that while fully automated domain ontology construction is possible, human supervision and early intervention typically improve efficiency and output quality as they lessen the effects of response randomness and the butterfly effect. We, therefore, also develop a web-based distillation assistant enabling supervision and flexible intervention at runtime. We hope our findings and tools could inspire future research toward revolutionizing the engineering of knowledge-based systems across application domains.