Learning How and What to Memorize: Cognition-Inspired Two-Stage Optimization for Evolving Memory

TL;DR

This paper introduces MemCoE, a cognition-inspired two-stage framework for optimizing long-term memory in language models, improving personalization and robustness over existing methods.

Contribution

It proposes a novel two-stage optimization approach inspired by cognition, combining memory guideline induction with guideline-aligned RL to enhance memory management.

Findings

Consistent improvements over strong baselines in personalization benchmarks.

Enhanced robustness, transferability, and efficiency demonstrated.

Effective handling of explicit and implicit preferences across various noise levels.

Abstract

Large language model (LLM) agents require long-term user memory for consistent personalization, but limited context windows hinder tracking evolving preferences over long interactions. Existing memory systems mainly rely on static, hand-crafted update rules; although reinforcement learning (RL)-based agents learn memory updates, sparse outcome rewards provide weak supervision, resulting in unstable long-horizon optimization. Drawing on memory schema theory and the functional division between prefrontal regions and hippocampus regions, we introduce MemCoE, a cognition-inspired two-stage optimization framework that learns how memory should be organized and what information to update. In the first stage, we propose Memory Guideline Induction to optimize a global guideline via contrastive feedback interpreted as textual gradients; in the second stage, Guideline-Aligned Memory Policy Optimization uses the induced guideline to define structured process rewards and performs multi-turn RL to learn a guideline-following memory evolution policy. We evaluate on three personalization memory benchmarks, covering explicit/implicit preference and different sizes and noise, and observe consistent improvements over strong baselines with favorable robustness, transferability, and efficiency.