MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants

TL;DR

MemSim is a Bayesian simulation framework that automatically generates reliable evaluation datasets to objectively assess the memory capabilities of LLM-based personal assistants, addressing a key challenge in the field.

Contribution

We introduce MemSim, a Bayesian simulator with a causal generation mechanism for automatic, scalable, and reliable memory evaluation of LLM-based agents.

Findings

MemSim effectively generates diverse evaluation datasets.

The MemDaily dataset enables benchmarking of memory mechanisms.

Our experiments demonstrate the simulator's reliability in assessing memory performance.

Abstract

LLM-based agents have been widely applied as personal assistants, capable of memorizing information from user messages and responding to personal queries. However, there still lacks an objective and automatic evaluation on their memory capability, largely due to the challenges in constructing reliable questions and answers (QAs) according to user messages. In this paper, we propose MemSim, a Bayesian simulator designed to automatically construct reliable QAs from generated user messages, simultaneously keeping their diversity and scalability. Specifically, we introduce the Bayesian Relation Network (BRNet) and a causal generation mechanism to mitigate the impact of LLM hallucinations on factual information, facilitating the automatic creation of an evaluation dataset. Based on MemSim, we generate a dataset in the daily-life scenario, named MemDaily, and conduct extensive experiments to assess the effectiveness of our approach. We also provide a benchmark for evaluating different memory mechanisms in LLM-based agents with the MemDaily dataset. To benefit the research community, we have released our project at https://github.com/nuster1128/MemSim.