Implicit Graph, Explicit Retrieval: Towards Efficient and Interpretable Long-horizon Memory for Large Language Models

TL;DR

LatentGraphMem is a novel memory system for large language models that combines implicit graph storage with explicit subgraph retrieval, improving efficiency, interpretability, and scalability in long-horizon reasoning tasks.

Contribution

It introduces a hybrid memory framework that stores graphs in latent space and retrieves explicit subgraphs, enhancing interpretability and performance in long-context applications.

Findings

Outperforms existing explicit and latent memory baselines on long-horizon benchmarks.

Enables parameter-efficient adaptation and scalable reasoning.

Maintains interpretability through explicit subgraph retrieval.

Abstract

Long-horizon applications increasingly require large language models (LLMs) to answer queries when relevant evidence is sparse and dispersed across very long contexts. Existing memory systems largely follow two paradigms: explicit structured memories offer interpretability but often become brittle under long-context overload, while latent memory mechanisms are efficient and stable yet difficult to inspect. We propose LatentGraphMem, a memory framework that combines implicit graph memory with explicit subgraph retrieval. LatentGraphMem stores a graph-structured memory in latent space for stability and efficiency, and exposes a task-specific subgraph retrieval interface that returns a compact symbolic subgraph under a fixed budget for downstream reasoning and human inspection. During training, an explicit graph view is materialized to interface with a frozen reasoner for question-answering supervision. At inference time, retrieval is performed in latent space and only the retrieved subgraph is externalized. Experiments on long-horizon benchmarks across multiple model scales show that LatentGraphMem consistently outperforms representative explicit-graph and latent-memory baselines, while enabling parameter-efficient adaptation and flexible scaling to larger reasoners without introducing large symbolic artifacts.