Personalized Socially Assistive Robots With End-to-End Speech-Language Models For Well-Being Support

TL;DR

This study explores the use of integrated end-to-end speech-language models in socially assistive robots to enhance real-time, personalized, and empathetic well-being support, based on a small user study.

Contribution

It introduces an SLM-enabled SAR dialogue system and evaluates its usability, identifying key limitations and areas for future improvement in real-time interaction and personalization.

Findings

Participants perceived the robot as empathetic and natural in dialogue.

Identified limitations in nonverbal behavior variability and synchronization.

Verbal responses were seen as generic and repetitive.

Abstract

Socially assistive robots (SARs) have shown great potential for supplementing well-being support. However, prior studies have found that existing dialogue pipelines for SARs remain limited in real-time latency, back-channeling, and personalized speech dialogue. Toward addressing these limitations, we propose using integrated end-to-end speech-language models (SLMs) with SARs. This work 1) evaluated the usability of an SLM-enabled SAR dialogue system through a small user study, and 2) identified remaining limitations through study user feedback to inform future improvements. We conducted a small within-participant user study with university students (N = 11) whose results showed that participants perceived an SLM-enabled SAR system as capable of providing empathetic feedback, natural turn-taking, back-channeling, and adaptive responses. We also found that participants reported the robot's nonverbal behaviors as lacking variability and synchronization with conversation, and the SLM's verbal feedback as generic and repetitive. These findings highlighted the need for real-time robot movement synchronized with conversation, improved prompting or fine-tuning to generate outputs better aligned with mental health practices, and more expressive, adaptive vocal generation.