Increasing the LLM Accuracy for Question Answering: Ontologies to the Rescue!

TL;DR

This paper introduces an ontology-based approach to improve the accuracy of LLM-powered question answering systems by detecting and repairing errors in generated queries, significantly increasing accuracy from 54% to 72%.

Contribution

The paper presents a novel ontology-based query check and repair method that enhances LLM question answering accuracy by leveraging knowledge graph ontologies.

Findings

Accuracy increased to 72% with the new approach.

Error rate reduced to 20%.

Additional 8% of results were 'I don't know' responses.

Abstract

There is increasing evidence that question-answering (QA) systems with Large Language Models (LLMs), which employ a knowledge graph/semantic representation of an enterprise SQL database (i.e. Text-to-SPARQL), achieve higher accuracy compared to systems that answer questions directly on SQL databases (i.e. Text-to-SQL). Our previous benchmark research showed that by using a knowledge graph, the accuracy improved from 16% to 54%. The question remains: how can we further improve the accuracy and reduce the error rate? Building on the observations of our previous research where the inaccurate LLM-generated SPARQL queries followed incorrect paths, we present an approach that consists of 1) Ontology-based Query Check (OBQC): detects errors by leveraging the ontology of the knowledge graph to check if the LLM-generated SPARQL query matches the semantic of ontology and 2) LLM Repair: use the error explanations with an LLM to repair the SPARQL query. Using the chat with the data benchmark, our primary finding is that our approach increases the overall accuracy to 72% including an additional 8% of "I don't know" unknown results. Thus, the overall error rate is 20%. These results provide further evidence that investing knowledge graphs, namely the ontology, provides higher accuracy for LLM powered question answering systems.