SocialMemBench: Are AI Memory Systems Ready for Social Group Settings?

TL;DR

SocialMemBench is a new benchmark designed to evaluate AI memory systems' ability to handle multi-party social group settings, revealing significant gaps in current systems' capabilities.

Contribution

The paper introduces SocialMemBench, a comprehensive benchmark for multi-party social group memory, and evaluates existing AI memory frameworks, highlighting their limitations.

Findings

Current memory systems cluster in the 0.12-0.18 range, below references of 0.345 and 0.369.

A full-context Gemini 2.5 Flash reaches only 0.721 on small networks, indicating difficulty.

Benchmark reveals measurable gaps in current AI memory systems for social group settings.

Abstract

Memory systems for AI assistants were built for single-user dialogue and fail characteristically when applied to multi-party social group settings. This gap matters for the social assistants being built today: group-acting agents embedded in chat platforms, and proactive personal-assistant agents whose holistic model of a user must include their social context. Existing memory benchmarks evaluate dyadic or workplace dialogue; none targets multi-party social groups, where memory must anchor facts in shared history rather than professional roles, separate group norms from individual exceptions, and correctly attribute even after member departure. We introduce SocialMemBench, a benchmark of human-verified synthetic social group networks across five archetypes (close friends, family, recreational, interest community, acquaintance network) and three group-size tiers (4-30 members), with 430 personas and 7,355 conversation turns, yielding 1,031 QA pairs across nine question categories. Each category isolates an architectural capability, and the five failure modes (single-stream conflation, temporal-state overwrite, entity merging at scale, missing cross-persona knowledge, norm-individual conflation) are testable hypotheses; our two research probes Subject-Mem and SMG provide evidence on two, three remain open. A full-context Gemini 2.5 Flash reference reaches only 0.721 against a blind-critic reasoning-model mean of 0.98 on small networks, indicating the benchmark is genuinely difficult even with complete access to the conversation. Across all 43 networks, the four open-source memory frameworks evaluated (Mem0, LangMem, Graphiti, Cognee) cluster in the 0.12-0.18 question-weighted range with overlapping 95% CIs, well below an uncompressed retrieval reference of 0.345 and a matched-answerer full-context reference of 0.369 (GPT-4o-mini). Current memory systems show a measurable gap.