HOLMES: Hyper-Relational Knowledge Graphs for Multi-hop Question Answering using LLMs

TL;DR

This paper introduces HOLMES, a context-aware, distilled knowledge graph that enhances multi-hop question answering with LLMs by reducing token usage and improving accuracy on benchmark datasets.

Contribution

HOLMES presents a novel query-relevant, compressed knowledge graph approach that outperforms existing methods in multi-hop question answering tasks.

Findings

HOLMES reduces token usage by up to 67% compared to state-of-the-art methods.

HOLMES achieves consistent improvements in EM, F1, BERTScore, and Human Eval metrics.

HOLMES outperforms baseline methods on HotpotQA and MuSiQue datasets.

Abstract

Given unstructured text, Large Language Models (LLMs) are adept at answering simple (single-hop) questions. However, as the complexity of the questions increase, the performance of LLMs degrade. We believe this is due to the overhead associated with understanding the complex question followed by filtering and aggregating unstructured information in the raw text. Recent methods try to reduce this burden by integrating structured knowledge triples into the raw text, aiming to provide a structured overview that simplifies information processing. However, this simplistic approach is query-agnostic and the extracted facts are ambiguous as they lack context. To address these drawbacks and to enable LLMs to answer complex (multi-hop) questions with ease, we propose to use a knowledge graph (KG) that is context-aware and is distilled to contain query-relevant information. The use of our compressed distilled KG as input to the LLM results in our method utilizing up to $67\%$ fewer tokens to represent the query relevant information present in the supporting documents, compared to the state-of-the-art (SoTA) method. Our experiments show consistent improvements over the SoTA across several metrics (EM, F1, BERTScore, and Human Eval) on two popular benchmark datasets (HotpotQA and MuSiQue).