CarMem: Enhancing Long-Term Memory in LLM Voice Assistants through Category-Bounding

TL;DR

This paper introduces CarMem, a long-term memory system for voice assistants that uses category-based preference extraction to improve personalization, privacy, and trust, validated on a synthetic dataset with high accuracy and reduced redundancies.

Contribution

The paper presents a novel category-bounding memory system for voice assistants that enhances preference retention, privacy, and transparency using LLMs, with a new synthetic dataset for benchmarking.

Findings

Preference extraction F1-score of .78 to .95

Reduces redundant preferences by 95%

Achieves retrieval accuracy of .87

Abstract

In today's assistant landscape, personalisation enhances interactions, fosters long-term relationships, and deepens engagement. However, many systems struggle with retaining user preferences, leading to repetitive user requests and disengagement. Furthermore, the unregulated and opaque extraction of user preferences in industry applications raises significant concerns about privacy and trust, especially in regions with stringent regulations like Europe. In response to these challenges, we propose a long-term memory system for voice assistants, structured around predefined categories. This approach leverages Large Language Models to efficiently extract, store, and retrieve preferences within these categories, ensuring both personalisation and transparency. We also introduce a synthetic multi-turn, multi-session conversation dataset (CarMem), grounded in real industry data, tailored to an in-car voice assistant setting. Benchmarked on the dataset, our system achieves an F1-score of .78 to .95 in preference extraction, depending on category granularity. Our maintenance strategy reduces redundant preferences by 95% and contradictory ones by 92%, while the accuracy of optimal retrieval is at .87. Collectively, the results demonstrate the system's suitability for industrial applications.