PersonalAI: A Systematic Comparison of Knowledge Graph Storage and Retrieval Approaches for Personalized LLM agents

TL;DR

This paper introduces a knowledge graph-based external memory framework for personalized LLM agents, enhancing long-term interaction capabilities and semantic-temporal reasoning.

Contribution

It presents a novel hybrid graph design and diverse retrieval mechanisms, improving personalization and reasoning in LLMs compared to existing methods.

Findings

Different memory and retrieval configurations optimize performance per task.

The framework effectively manages temporal dependencies and contradictions.

Evaluation on multiple benchmarks demonstrates robustness and adaptability.

Abstract

Personalizing language models by effectively incorporating user interaction history remains a central challenge in the development of adaptive AI systems. While large language models (LLMs), combined with Retrieval-Augmented Generation (RAG), have improved factual accuracy, they often lack structured memory and fail to scale in complex, long-term interactions. To address this, we propose a flexible external memory framework based on a knowledge graph that is constructed and updated automatically by the LLM. Building upon the AriGraph architecture, we introduce a novel hybrid graph design that supports both standard edges and two types of hyper-edges, enabling rich and dynamic semantic and temporal representations. Our framework also supports diverse retrieval mechanisms, including A*, WaterCircles traversal, beam search, and hybrid methods, making it adaptable to different datasets and LLM capacities. We evaluate our system on TriviaQA, HotpotQA, DiaASQ benchmarks and demonstrate that different memory and retrieval configurations yield optimal performance depending on the task. Additionally, we extend the DiaASQ benchmark with temporal annotations and internally contradictory statements, showing that our system remains robust and effective in managing temporal dependencies and context-aware reasoning