Does Your Voice Assistant Remember? Analyzing Conversational Context Recall and Utilization in Voice Interaction Models

TL;DR

This paper evaluates how well open-source voice interaction models recall and utilize past conversations, revealing significant limitations especially in speech-based models and suggesting directions for improvement.

Contribution

It introduces ContextDialog, a benchmark for assessing conversational memory in open-source models, and provides a systematic analysis of their recall capabilities.

Findings

Speech-based models struggle more than text-based models in recalling past utterances.

Retrieval-augmented generation does not fully solve memory recall issues.

Open-source models have notable limitations in conversational context retention.

Abstract

Recent advancements in multi-turn voice interaction models have improved user-model communication. However, while closed-source models effectively retain and recall past utterances, whether open-source models share this ability remains unexplored. To fill this gap, we systematically evaluate how well open-source interaction models utilize past utterances using ContextDialog, a benchmark we proposed for this purpose. Our findings show that speech-based models have more difficulty than text-based ones, especially when recalling information conveyed in speech, and even with retrieval-augmented generation, models still struggle with questions about past utterances. These insights highlight key limitations in open-source models and suggest ways to improve memory retention and retrieval robustness.