Knowledge Graph Structure as Prompt: Improving Small Language Models Capabilities for Knowledge-based Causal Discovery

TL;DR

This paper introduces KG Structure as Prompt, a method that enhances small language models' ability to perform knowledge-based causal discovery by integrating knowledge graph structures into prompt-based learning, outperforming larger models and traditional fine-tuning.

Contribution

The paper presents a novel prompt-based approach that incorporates knowledge graph structures to improve small language models' causal discovery capabilities, especially in few-shot settings.

Findings

SLMs with KG Structure as Prompt outperform baselines

Approach surpasses full-data fine-tuning methods

SLMs demonstrate strong potential with knowledge graphs

Abstract

Causal discovery aims to estimate causal structures among variables based on observational data. Large Language Models (LLMs) offer a fresh perspective to tackle the causal discovery problem by reasoning on the metadata associated with variables rather than their actual data values, an approach referred to as knowledge-based causal discovery. In this paper, we investigate the capabilities of Small Language Models (SLMs, defined as LLMs with fewer than 1 billion parameters) with prompt-based learning for knowledge-based causal discovery. Specifically, we present KG Structure as Prompt, a novel approach for integrating structural information from a knowledge graph, such as common neighbor nodes and metapaths, into prompt-based learning to enhance the capabilities of SLMs. Experimental results on three types of biomedical and open-domain datasets under few-shot settings demonstrate the effectiveness of our approach, surpassing most baselines and even conventional fine-tuning approaches trained on full datasets. Our findings further highlight the strong capabilities of SLMs: in combination with knowledge graphs and prompt-based learning, SLMs demonstrate the potential to surpass LLMs with larger number of parameters. Our code and datasets are available on GitHub.