UniOQA: A Unified Framework for Knowledge Graph Question Answering with Large Language Models

TL;DR

UniOQA introduces a unified framework leveraging large language models and novel algorithms to improve question answering accuracy over the extensive Chinese knowledge graph OwnThink, achieving state-of-the-art results.

Contribution

The paper presents a novel unified framework combining LLM fine-tuning, entity-relation replacement, and retrieval-augmented generation for knowledge graph question answering.

Findings

Achieves new state-of-the-art accuracy on OwnThink benchmark

Demonstrates improved representation capacity with fine-tuned LLMs

Validates effectiveness of the combined workflows through ablation studies

Abstract

OwnThink stands as the most extensive Chinese open-domain knowledge graph introduced in recent times. Despite prior attempts in question answering over OwnThink (OQA), existing studies have faced limitations in model representation capabilities, posing challenges in further enhancing overall accuracy in question answering. In this paper, we introduce UniOQA, a unified framework that integrates two complementary parallel workflows. Unlike conventional approaches, UniOQA harnesses large language models (LLMs) for precise question answering and incorporates a direct-answer-prediction process as a cost-effective complement. Initially, to bolster representation capacity, we fine-tune an LLM to translate questions into the Cypher query language (CQL), tackling issues associated with restricted semantic understanding and hallucinations. Subsequently, we introduce the Entity and Relation Replacement algorithm to ensure the executability of the generated CQL. Concurrently, to augment overall accuracy in question answering, we further adapt the Retrieval-Augmented Generation (RAG) process to the knowledge graph. Ultimately, we optimize answer accuracy through a dynamic decision algorithm. Experimental findings illustrate that UniOQA notably advances SpCQL Logical Accuracy to 21.2% and Execution Accuracy to 54.9%, achieving the new state-of-the-art results on this benchmark. Through ablation experiments, we delve into the superior representation capacity of UniOQA and quantify its performance breakthrough.