Aligning Large Language Models to a Domain-specific Graph Database for NL2GQL

TL;DR

This paper presents a pipeline that uses ChatGPT to generate domain-specific NL-GQL data pairs, fine-tunes LLMs for NL2GQL tasks, and emphasizes the importance of schema relevance, achieving significant performance improvements in finance and medicine domains.

Contribution

The paper introduces a novel method for generating domain-specific NL-GQL data using ChatGPT and schema extraction, enhancing LLM alignment for NL2GQL tasks.

Findings

Significant improvements in EM and EX metrics over baselines.

Effective use of schema relevance for accurate GQL generation.

Successful application in finance and medicine domains.

Abstract

Graph Databases (Graph DB) find extensive application across diverse domains such as finance, social networks, and medicine. Yet, the translation of Natural Language (NL) into the Graph Query Language (GQL), referred to as NL2GQL, poses significant challenges owing to its intricate and specialized nature. Some approaches have sought to utilize Large Language Models (LLMs) to address analogous tasks like text2SQL. Nonetheless, in the realm of NL2GQL tasks tailored to a particular domain, the absence of domain-specific NL-GQL data pairs adds complexity to aligning LLMs with the graph DB. To tackle this challenge, we present a well-defined pipeline. Initially, we utilize ChatGPT to generate NL-GQL data pairs, leveraging the provided graph DB with self-instruction. Subsequently, we employ the generated data to fine-tune LLMs, ensuring alignment between LLMs and the graph DB. Moreover, we find the importance of relevant schema in efficiently generating accurate GQLs. Thus, we introduce a method to extract relevant schema as the input context. We evaluate our method using two carefully constructed datasets derived from graph DBs in the finance and medicine domains, named FinGQL and MediGQL. Experimental results reveal that our approach significantly outperforms a set of baseline methods, with improvements of 5.90 and 6.36 absolute points on EM, and 6.00 and 7.09 absolute points on EX for FinGQL and MediGQL, respectively.