LatentMem: Customizing Latent Memory for Multi-Agent Systems

TL;DR

LatentMem is a learnable, role-aware multi-agent memory system that enhances performance by creating compact, agent-specific memories, addressing homogenization and overload issues in multi-agent systems.

Contribution

We introduce LatentMem, a novel memory framework with experience banks and a memory composer, plus LMPO for task-driven optimization, improving multi-agent memory customization.

Findings

Achieves up to 19.36% performance improvement over vanilla systems.

Outperforms existing memory architectures across benchmarks.

Does not require modifications to underlying frameworks.

Abstract

Large language model (LLM)-powered multi-agent systems (MAS) demonstrate remarkable collective intelligence, wherein multi-agent memory serves as a pivotal mechanism for continual adaptation. However, existing multi-agent memory designs remain constrained by two fundamental bottlenecks: (i) memory homogenization arising from the absence of role-aware customization, and (ii) information overload induced by excessively fine-grained memory entries. To address these limitations, we propose LatentMem, a learnable multi-agent memory framework designed to customize agent-specific memories in a token-efficient manner. Specifically, LatentMem comprises an experience bank that stores raw interaction trajectories in a lightweight form, and a memory composer that synthesizes compact latent memories conditioned on retrieved experience and agent-specific contexts. Further, we introduce Latent Memory Policy Optimization (LMPO), which propagates task-level optimization signals through latent memories to the composer, encouraging it to produce compact and high-utility representations. Extensive experiments across diverse benchmarks and mainstream MAS frameworks show that LatentMem achieves a performance gain of up to $19.36$% over vanilla settings and consistently outperforms existing memory architectures, without requiring any modifications to the underlying frameworks.