Efficient Knowledge Infusion via KG-LLM Alignment

TL;DR

This paper introduces a method to improve large language models' use of domain-specific knowledge graphs by constructing tailored graphs with minimal data and aligning LLMs more effectively, leading to better biomedical question-answering performance.

Contribution

It presents a novel approach combining small labeled samples and large corpus to build domain-specific knowledge graphs and a three-stage alignment strategy to enhance LLM utilization of these graphs.

Findings

Outperforms existing baselines on biomedical QA datasets

Effective knowledge graph construction with limited samples

Enhanced LLM knowledge compliance through alignment strategy

Abstract

To tackle the problem of domain-specific knowledge scarcity within large language models (LLMs), knowledge graph-retrievalaugmented method has been proven to be an effective and efficient technique for knowledge infusion. However, existing approaches face two primary challenges: knowledge mismatch between public available knowledge graphs and the specific domain of the task at hand, and poor information compliance of LLMs with knowledge graphs. In this paper, we leverage a small set of labeled samples and a large-scale corpus to efficiently construct domain-specific knowledge graphs by an LLM, addressing the issue of knowledge mismatch. Additionally, we propose a three-stage KG-LLM alignment strategyto enhance the LLM's capability to utilize information from knowledge graphs. We conduct experiments with a limited-sample setting on two biomedical question-answering datasets, and the results demonstrate that our approach outperforms existing baselines.