From Experience to Strategy: Empowering LLM Agents with Trainable Graph Memory

TL;DR

This paper introduces a trainable graph memory framework for LLM agents that enhances their reasoning by structuring experiences into decision paths and optimizing strategic meta-cognition through reinforcement learning.

Contribution

It presents a novel, trainable, multi-layered graph memory that improves LLM agent reasoning and adaptability by integrating structured experience and meta-cognitive strategies.

Findings

Improves strategic reasoning performance of LLM agents.

Enhances generalization and robustness during RL training.

Provides interpretable decision paths and meta-cognitive strategies.

Abstract

Large Language Models (LLMs) based agents have demonstrated remarkable potential in autonomous task-solving across complex, open-ended environments. A promising approach for improving the reasoning capabilities of LLM agents is to better utilize prior experiences in guiding current decisions. However, LLMs acquire experience either through implicit memory via training, which suffers from catastrophic forgetting and limited interpretability, or explicit memory via prompting, which lacks adaptability. In this paper, we introduce a novel agent-centric, trainable, multi-layered graph memory framework and evaluate how context memory enhances the ability of LLMs to utilize parametric information. The graph abstracts raw agent trajectories into structured decision paths in a state machine and further distills them into high-level, human-interpretable strategic meta-cognition. In order to make memory adaptable, we propose a reinforcement-based weight optimization procedure that estimates the empirical utility of each meta-cognition based on reward feedback from downstream tasks. These optimized strategies are then dynamically integrated into the LLM agent's training loop through meta-cognitive prompting. Empirically, the learnable graph memory delivers robust generalization, improves LLM agents' strategic reasoning performance, and provides consistent benefits during Reinforcement Learning (RL) training.