From Documents to Dialogue: Building KG-RAG Enhanced AI Assistants

TL;DR

This paper introduces a novel KG-RAG framework that enhances AI assistants' ability to retrieve and utilize private enterprise data through a knowledge graph, significantly improving response accuracy and relevance.

Contribution

We propose a new method for constructing high-quality, low-noise knowledge graphs integrated with RAG to improve AI assistant responses over private data sources.

Findings

Response relevance increased by over 50%

Fully relevant answers improved by 88%

Effective filtering reduces noise in knowledge graphs

Abstract

The Adobe Experience Platform AI Assistant is a conversational tool that enables organizations to interact seamlessly with proprietary enterprise data through a chatbot. However, due to access restrictions, Large Language Models (LLMs) cannot retrieve these internal documents, limiting their ability to generate accurate zero-shot responses. To overcome this limitation, we use a Retrieval-Augmented Generation (RAG) framework powered by a Knowledge Graph (KG) to retrieve relevant information from external knowledge sources, enabling LLMs to answer questions over private or previously unseen document collections. In this paper, we propose a novel approach for building a high-quality, low-noise KG. We apply several techniques, including incremental entity resolution using seed concepts, similarity-based filtering to deduplicate entries, assigning confidence scores to entity-relation pairs to filter for high-confidence pairs, and linking facts to source documents for provenance. Our KG-RAG system retrieves relevant tuples, which are added to the user prompts context before being sent to the LLM generating the response. Our evaluation demonstrates that this approach significantly enhances response relevance, reducing irrelevant answers by over 50% and increasing fully relevant answers by 88% compared to the existing production system.