Optimizing the Interface Between Knowledge Graphs and LLMs for Complex Reasoning

TL;DR

This paper investigates hyperparameter optimization in systems combining Knowledge Graphs and Large Language Models, demonstrating that targeted tuning can significantly improve multi-hop question answering performance.

Contribution

It introduces a systematic approach to hyperparameter tuning in KG-LLM systems and evaluates its impact across multiple QA benchmarks.

Findings

Hyperparameter tuning yields meaningful performance gains.

Performance improvements vary across datasets and metrics.

Standard evaluation metrics have limitations in capturing true system performance.

Abstract

Integrating Large Language Models (LLMs) with Knowledge Graphs (KGs) results in complex systems with numerous hyperparameters that directly affect performance. While such systems are increasingly common in retrieval-augmented generation, the role of systematic hyperparameter optimization remains underexplored. In this paper, we study this problem in the context of Cognee, a modular framework for end-to-end KG construction and retrieval. Using three multi-hop QA benchmarks (HotPotQA, TwoWikiMultiHop, and MuSiQue) we optimize parameters related to chunking, graph construction, retrieval, and prompting. Each configuration is scored using established metrics (exact match, F1, and DeepEval's LLM-based correctness metric). Our results demonstrate that meaningful gains can be achieved through targeted tuning. While the gains are consistent, they are not uniform, with performance varying across datasets and metrics. This variability highlights both the value of tuning and the limitations of standard evaluation measures. While demonstrating the immediate potential of hyperparameter tuning, we argue that future progress will depend not only on architectural advances but also on clearer frameworks for optimization and evaluation in complex, modular systems.