Fine-Mem: Fine-Grained Feedback Alignment for Long-Horizon Memory Management

TL;DR

Fine-Mem introduces a novel fine-grained feedback mechanism for memory management in large language models, improving credit assignment and task performance in long-horizon tasks.

Contribution

It proposes a unified framework with chunk-level rewards and evidence-anchored reward attribution to enhance memory operation guidance.

Findings

Outperforms strong baselines on Memalpha and MemoryAgentBench.

Achieves higher success rates across various sub-tasks.

Demonstrates strong generalization across models and backbones.

Abstract

Effective memory management is essential for large language model agents to navigate long-horizon tasks. Recent research has explored using Reinforcement Learning to develop specialized memory manager agents. However, existing approaches rely on final task performance as the primary reward, which results in severe reward sparsity and ineffective credit assignment, providing insufficient guidance for individual memory operations. To this end, we propose Fine-Mem, a unified framework designed for fine-grained feedback alignment. First, we introduce a Chunk-level Step Reward to provide immediate step-level supervision via auxiliary chunk-specific question answering tasks. Second, we devise Evidence-Anchored Reward Attribution to redistribute global rewards by anchoring credit to key memory operations, based on the specific memory items utilized as evidence in reasoning. Together, these components enable stable policy optimization and align local memory operations with the long-term utility of memory. Experiments on Memalpha and MemoryAgentBench demonstrate that Fine-Mem consistently outperforms strong baselines, achieving superior success rates across various sub-tasks. Further analysis reveals its adaptability and strong generalization capabilities across diverse model configurations and backbones.