Cost and accuracy of long-term memory in Distributed Multi-Agent Systems based on Large Language Models

TL;DR

This paper evaluates long-term memory solutions in distributed multi-agent systems based on large language models, focusing on efficiency and cost-effectiveness under network constraints, and finds mem0 to be superior in performance.

Contribution

Introduces a systematic comparison of mem0 and Graphiti memory frameworks using a new benchmark under various network conditions, highlighting efficiency advantages of mem0.

Findings

mem0 outperforms Graphiti in efficiency and resource usage

Accuracy differences between mem0 and Graphiti are not statistically significant

mem0 is identified as the optimal cost-accuracy balance using Pareto efficiency

Abstract

Distributed multi-agent systems (DMAS) based on large language models (LLMs) enable collaborative intelligence while preserving data privacy. However, systematic evaluations of long-term memory under network constraints are limited. This study introduces a flexible testbed to compare mem0, a vector-based memory framework, and Graphiti, a graph-based knowledge graph, using the LoCoMo long-context benchmark. Experiments were conducted under unconstrained and constrained network conditions, measuring computational, financial, and accuracy metrics. Results indicate mem0 significantly outperforms Graphiti in efficiency, featuring faster loading times, lower resource consumption, and minimal network overhead. Crucially, accuracy differences were not statistically significant. Applying a statistical Pareto efficiency framework, mem0 is identified as the optimal choice, balancing cost and accuracy in DMAS.