AFA: Identity-Aware Memory for Preventing Persona Confusion in Multi-User Dialogue

TL;DR

This paper introduces AFA, a framework combining voice identification and per-user memory to prevent persona confusion in multi-user dialogue systems, improving personalization and trust.

Contribution

The paper presents AFA, a modular identity-aware framework with a synthetic dataset and evaluation protocols, demonstrating improved persona attribution accuracy in multi-user dialogue.

Findings

AFA reduces persona confusion from 35.7% to 61.3% PAA.

Fine-tuned LLaMA-2-70B achieves highest performance on PAT.

Human evaluations show increased perceived personalization.

Abstract

When multiple people share a single voice assistant, the system conflates their histories: one resident's preferences can leak into another's responses, eroding utility and trust. We call this failure mode persona confusion, and we show it is a measurable problem in today's single-user dialogue systems when deployed in shared environments. We present the Adaptive Friend Agent (AFA), a modular framework that combines voice-based speaker identification with per-user memory stores to enable identity-aware, personalized dialogue across multiple users. To support training and evaluation, we construct PAT (Personalized Agent chaT), a synthetic dataset of 58,289 persona-grounded dialogue turns spanning 133 user profiles and 12 real-world scenarios. We evaluate AFA across five LLM back-ends in a standard response-quality benchmark, with a LLaMA-2-70B model fine-tuned on PAT achieving the highest overall performance. To directly measure persona confusion prevention, we introduce an interleaved multi-user evaluation protocol with a novel metric, Persona Attribution Accuracy (PAA), demonstrating that identity-aware routing improves PAA from 35.7% to 61.3%. Human evaluation confirms annotators perceive significantly higher personalization in routing-enabled responses. Our results establish that identity-aware user routing is the critical component for preventing persona confusion in multi-user conversational systems.