ReFactX: Scalable Reasoning with Reliable Facts via Constrained Generation

TL;DR

ReFactX introduces a scalable, efficient method for LLMs to access external knowledge through constrained generation with a prefix-tree index, improving reliability without complex pipelines.

Contribution

It presents a novel constrained generation approach using a prefix-tree index for external knowledge access, eliminating the need for retrievers or auxiliary models.

Findings

Scales to large knowledge bases with 800 million facts

Achieves effective question answering results

Minimal generation-time overhead

Abstract

Knowledge gaps and hallucinations are persistent challenges for Large Language Models (LLMs), which generate unreliable responses when lacking the necessary information to fulfill user instructions. Existing approaches, such as Retrieval-Augmented Generation (RAG) and tool use, aim to address these issues by incorporating external knowledge. Yet, they rely on additional models or services, resulting in complex pipelines, potential error propagation, and often requiring the model to process a large number of tokens. In this paper, we present a scalable method that enables LLMs to access external knowledge without depending on retrievers or auxiliary models. Our approach uses constrained generation with a pre-built prefix-tree index. Triples from a Knowledge Graph are verbalized in textual facts, tokenized, and indexed in a prefix tree for efficient access. During inference, to acquire external knowledge, the LLM generates facts with constrained generation which allows only sequences of tokens that form an existing fact. We evaluate our proposal on Question Answering and show that it scales to large knowledge bases (800 million facts), adapts to domain-specific data, and achieves effective results. These gains come with minimal generation-time overhead. ReFactX code is available at https://github.com/rpo19/ReFactX.